返回列表
PhotoMaker X Ascend快速启动
2024-01-24
浏览量：1831
# Photomaker
PhotoMaker是一个由Tencent分享的StableDiffusion的小玩具，在HuggingFace的官网上也挂了好一阵子，最近有时间在Ascend上做一下相关适配，将过程中碰到的一些问题记录在此。
[HuggingFace链接](https://huggingface.co/TencentARC/PhotoMaker)
[Github链接](https://github.com/TencentARC/PhotoMaker)
# 推理代码
按照README里面提供的推理代码，我们先初步输出一个推理代码
```
import torch
import torch_npu
import os
from diffusers.utils import load_image
from diffusers import EulerDiscreteScheduler
from photomaker.pipeline import PhotoMakerStableDiffusionXLPipeline
### Load base model
device = "npu:0"
base_model_path = "/home/ma-user/work/aicc/l00564131/models/stable-diffusion-xl-base-1.0"
photomaker_path = "/home/ma-user/work/aicc/l00564131/models/TencentARC/PhotoMaker/photomaker-v1.bin"
pipe = PhotoMakerStableDiffusionXLPipeline.from_pretrained(
base_model_path, # can change to any base model based on SDXL
torch_dtype=torch.bfloat16,
use_safetensors=True,
variant="fp16"
).to(device)
### Load PhotoMaker checkpoint
pipe.load_photomaker_adapter(
os.path.dirname(photomaker_path),
subfolder="",
weight_name=os.path.basename(photomaker_path),
trigger_word="img" # define the trigger word
)
pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config)
### Also can cooperate with other LoRA modules
# pipe.load_lora_weights(os.path.dirname(lora_path), weight_name=lora_model_name, adapter_name="xl_more_art-full")
# pipe.set_adapters(["photomaker", "xl_more_art-full"], adapter_weights=[1.0, 0.5])
pipe.fuse_lora()
### define the input ID images
input_folder_name = './examples/newton_man'
image_basename_list = os.listdir(input_folder_name)
image_path_list = sorted([os.path.join(input_folder_name, basename) for basename in image_basename_list])
input_id_images = []
for image_path in image_path_list:
input_id_images.append(load_image(image_path))
# Note that the trigger word `img` must follow the class word for personalization
prompt = "a half-body portrait of a man img wearing the sunglasses in Iron man suit, best quality"
negative_prompt = "(asymmetry, worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), open mouth, grayscale"
num_steps = 50
generator = torch.Generator(device=device).manual_seed(42)
images = pipe(
prompt=prompt,
input_id_images=input_id_images,
negative_prompt=negative_prompt,
num_images_per_prompt=1,
num_inference_steps=num_steps,
start_merge_step=10,
generator=generator,
).images[0]
gen_images.save('out_photomaker.png')
```
执行上述代码，会出现以下的报错
```
Loading PhotoMaker components [1] id_encoder from [/home/ma-user/work/aicc/l00564131/models/TencentARC/PhotoMaker]...
Loading PhotoMaker components [2] lora_weights from [/home/ma-user/work/aicc/l00564131/models/TencentARC/PhotoMaker]
E39999: Inner Error!
E39999 The error from device(chipId:0, dieId:0), serial number is 3, an exception occurred during AICPU execution, stream_id:2, task_id:7572, errcode:21008, msg:inner error.[FUNC:ProcessStarsAicpuErrorInfo][FILE:device_error_proc.cc][LINE:1188]
TraceBack (most recent call last):
Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:task_info.cc][LINE:1612]
AICPU Kernel task happen error, retCode=0x2a.[FUNC:GetError][FILE:stream.cc][LINE:1483]
Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=7572, errorCode=2a.[FUNC:PrintAicpuErrorInfo][FILE:task_info.cc][LINE:1457]
Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=7572, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1483]
rtStreamSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
Traceback (most recent call last):
File "/home/ma-user/work/aicc/l00564131/photomaker/PhotoMaker/gogogo.py", line 52, in
images = pipe(
File "/home/ma-user/anaconda3/envs/chatglm3-demo/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/ma-user/work/aicc/l00564131/photomaker/PhotoMaker/photomaker/pipeline.py", line 386, in __call__
self.scheduler.set_timesteps(num_inference_steps, device=device)
File "/home/ma-user/anaconda3/envs/chatglm3-demo/lib/python3.10/site-packages/diffusers/schedulers/scheduling_euler_discrete.py", line 335, in set_timesteps
sigmas = torch.from_numpy(sigmas).to(dtype=torch.float32, device=device)
RuntimeError: ACL stream synchronize failed, error code:507018
[W NPUStream.cpp:372] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
EH9999 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeDevice)
[W NPUStream.cpp:372] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
EH9999 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeDevice)
/home/ma-user/anaconda3/envs/chatglm3-demo/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up
_warnings.warn(warn_message, ResourceWarning)
(chatglm3-demo) [root@6ea3ed6b883e PhotoMaker]#
```
说实话，这部分报错日志能够体现的内容非常少，所以我们还是需要打开原始的host日志进行分析。我们可以在`/root/ascend/log`下面获取到设备运行时的日志，在此目录下执行`grep -rn ERROR ./`有
```
(chatglm3-demo) [root@6ea3ed6b883e log]# grep -rn ERROR ./
./debug/plog/plog-1151919_20240124104533663.log:27:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.034.295 [engine.cc:3775]1151919 ProcLogicCqReport:[FINAL][FINAL]Task run failed, device_id=0, stream_id=2, task_id=7572, sqe_type=1(aicpu), errType=0x1(task exception), sqSwStatus=0x5210
./debug/plog/plog-1151919_20240124104533663.log:28:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.194 [runtime.cc:4347]1151919 SetWatchDogDevStatus:[FINAL][FINAL]There is errInfo of devId=0, tsId=0
./debug/plog/plog-1151919_20240124104533663.log:29:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.228 [device_error_proc.cc:1188]1151919 ProcessStarsAicpuErrorInfo:[FINAL][FINAL]report error module_type=0, module_name=E39999
./debug/plog/plog-1151919_20240124104533663.log:30:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.234 [device_error_proc.cc:1188]1151919 ProcessStarsAicpuErrorInfo:[FINAL][FINAL]The error from device(chipId:0, dieId:0), serial number is 3, an exception occurred during AICPU execution, stream_id:2, task_id:7572, errcode:21008, msg:inner error.
./debug/plog/plog-1151919_20240124104533663.log:31:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.288 [stream.cc:3116]1151919 EnterFailureAbort:[FINAL][FINAL]stream_id=2 enter failure abort.
./debug/plog/plog-1151919_20240124104533663.log:32:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.298 [task_info.cc:1718]1151919 SetStarsResultForDavinciTask:[FINAL][FINAL]AICPU Kernel task happen error, retCode=0x2a.
./debug/plog/plog-1151919_20240124104533663.log:33:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.307 [task_info.cc:1612]1151919 PreCheckTaskErr:[FINAL][FINAL]report error module_type=5, module_name=EZ9999
./debug/plog/plog-1151919_20240124104533663.log:34:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.312 [task_info.cc:1612]1151919 PreCheckTaskErr:[FINAL][FINAL]Kernel task happen error, retCode=0x2a, [aicpu exception].
./debug/plog/plog-1151919_20240124104533663.log:35:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.325 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception].
./debug/plog/plog-1151919_20240124104533663.log:36:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.330 [stream.cc:1483]1151919 GetError:[FINAL][FINAL]report error module_type=0, module_name=E39999
./debug/plog/plog-1151919_20240124104533663.log:37:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.334 [stream.cc:1483]1151919 GetError:[FINAL][FINAL]AICPU Kernel task happen error, retCode=0x2a.
./debug/plog/plog-1151919_20240124104533663.log:38:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.344 [task_info.cc:1457]1151919 PrintAicpuErrorInfo:[FINAL][FINAL]report error module_type=0, module_name=E39999
./debug/plog/plog-1151919_20240124104533663.log:39:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.348 [task_info.cc:1457]1151919 PrintAicpuErrorInfo:[FINAL][FINAL]Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=7572, errorCode=2a.
./debug/plog/plog-1151919_20240124104533663.log:40:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.381 [task_info.cc:1467]1151919 PrintAicpuErrorInfo:[FINAL][FINAL]Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=7572, fault op_name=
./debug/plog/plog-1151919_20240124104533663.log:41:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.414 [engine.cc:3646]1151919 StarsResumeRtsq:[FINAL][FINAL]stop scheduling in abort failure mode: stream_id=2, sq_id=2, sq_head=3476, task_id=7572, taskType=1.
./debug/plog/plog-1151919_20240124104533663.log:42:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.429 [stream.cc:2904]1151919 StarsWaitForTask:[FINAL][FINAL]Task Wait: device_id=0, stream_id=2 is Abort, runningState=0
./debug/plog/plog-1151919_20240124104533663.log:43:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.434 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception].
./debug/plog/plog-1151919_20240124104533663.log:44:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.438 [stream.cc:1483]1151919 GetError:[FINAL][FINAL]report error module_type=0, module_name=E39999
./debug/plog/plog-1151919_20240124104533663.log:45:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.442 [stream.cc:1483]1151919 GetError:[FINAL][FINAL]Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=7572, fault op_name=
./debug/plog/plog-1151919_20240124104533663.log:46:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.452 [logger.cc:472]1151919 StreamSynchronize:[FINAL][FINAL]Stream synchronize failed, stream_id=2
./debug/plog/plog-1151919_20240124104533663.log:47:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.468 [api_c.cc:787]1151919 rtStreamSynchronize:[FINAL][FINAL]ErrCode=507018, desc=[aicpu exception], InnerCode=0x715002a
./debug/plog/plog-1151919_20240124104533663.log:48:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.472 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888
./debug/plog/plog-1151919_20240124104533663.log:49:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.479 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]rtStreamSynchronize execute failed, reason=[aicpu exception]
./debug/plog/plog-1151919_20240124104533663.log:50:[ERROR] ASCENDCL(1151919,python):2024-01-24-10:46:23.036.489 [stream.cpp:121]1151919 aclrtSynchronizeStream: [FINAL][FINAL]synchronize stream failed, runtime result = 507018
./debug/plog/plog-1151919_20240124104533663.log:51:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.404 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception].
./debug/plog/plog-1151919_20240124104533663.log:52:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.415 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception].
./debug/plog/plog-1151919_20240124104533663.log:53:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.420 [context.cc:306]1151919 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x715002a.
./debug/plog/plog-1151919_20240124104533663.log:54:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.504 [context.cc:311]1151919 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x715002a.
./debug/plog/plog-1151919_20240124104533663.log:55:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.509 [logger.cc:862]1151919 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed.
./debug/plog/plog-1151919_20240124104533663.log:56:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.520 [api_c.cc:1762]1151919 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=507018, desc=[aicpu exception], InnerCode=0x715002a
./debug/plog/plog-1151919_20240124104533663.log:57:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.525 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888
./debug/plog/plog-1151919_20240124104533663.log:58:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.530 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[aicpu exception]
./debug/plog/plog-1151919_20240124104533663.log:59:[ERROR] ASCENDCL(1151919,python):2024-01-24-10:46:23.050.556 [device.cpp:285]1151919 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 507018.
./debug/plog/plog-1151919_20240124104533663.log:60:[ERROR] APP(1151919,python):2024-01-24-10:46:23.051.537 [log_inner.cpp:76]1151919 $(notdir $(abspath $<)):THPModule_npu_shutdown:58: "[PTA]:"NPU shutdown synchronize device failed.""
./debug/plog/plog-1151919_20240124104533663.log:61:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.571 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception].
./debug/plog/plog-1151919_20240124104533663.log:62:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.577 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception].
./debug/plog/plog-1151919_20240124104533663.log:63:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.581 [context.cc:306]1151919 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x715002a.
./debug/plog/plog-1151919_20240124104533663.log:64:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.596 [context.cc:311]1151919 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x715002a.
./debug/plog/plog-1151919_20240124104533663.log:65:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.601 [logger.cc:862]1151919 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed.
./debug/plog/plog-1151919_20240124104533663.log:66:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.608 [api_c.cc:1762]1151919 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=507018, desc=[aicpu exception], InnerCode=0x715002a
./debug/plog/plog-1151919_20240124104533663.log:67:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.612 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888
./debug/plog/plog-1151919_20240124104533663.log:68:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.617 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[aicpu exception]
./debug/plog/plog-1151919_20240124104533663.log:69:[ERROR] ASCENDCL(1151919,python):2024-01-24-10:46:23.051.637 [device.cpp:285]1151919 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 507018.
./debug/device-0/device-1151919_20240124104548927.log:77:[ERROR] AICPU(14937,aicpu_scheduler):2024-01-24-10:46:22.212.021 [masked_scatter.cc:96][AICPU][Compute:96][tid:14948]:[MaskedScatter] Data type of input is not support, input data type is [DT_BFLOAT16].
./debug/device-0/device-1151919_20240124104548927.log:78:[ERROR] CCECPU(14937,aicpu_scheduler):2024-01-24-10:46:22.212.044 [ae_kernel_lib_aicpu.cc:277][TransformKernelErrorCode][tid:14948][AICPU_PROCESSER] call aicpu api RunCpuKernel in libcpu_kernels.so failed, ret:4294967295.
./debug/device-0/device-1151919_20240124104548927.log:79:[ERROR] CCECPU(14937,aicpu_scheduler):2024-01-24-10:46:22.212.054 [aicpusd_event_process.cpp:1448][ExecuteTsKernelTask][tid:14948] Aicpu engine process failed, result[-1], opName[MaskedScatter].
(chatglm3-demo) [root@6ea3ed6b883e log]#
```
上面的很多信息并不重要，直接看下面，不难发现在./debug/device-0/日志下面有一条很关键的**Data type of input is not support, input data type is [DT_BFLOAT16].**，我们在原始的推理代码中，按照官方建议设置的数据类型为BF16，那么我们将这部分代码修改为HALF（FP16）即可。
最终使用的推理代码为
```
import torch
import torch_npu
import os
from diffusers.utils import load_image
from diffusers import EulerDiscreteScheduler
from photomaker.pipeline import PhotoMakerStableDiffusionXLPipeline
### Load base model
device = "npu:0"
base_model_path = "/home/ma-user/work/aicc/l00564131/models/stable-diffusion-xl-base-1.0"
photomaker_path = "/home/ma-user/work/aicc/l00564131/models/TencentARC/PhotoMaker/photomaker-v1.bin"
pipe = PhotoMakerStableDiffusionXLPipeline.from_pretrained(
base_model_path, # can change to any base model based on SDXL
torch_dtype=torch.half,
use_safetensors=True,
variant="fp16"
).to(device)
### Load PhotoMaker checkpoint
pipe.load_photomaker_adapter(
os.path.dirname(photomaker_path),
subfolder="",
weight_name=os.path.basename(photomaker_path),
trigger_word="img" # define the trigger word
)
pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config)
### Also can cooperate with other LoRA modules
# pipe.load_lora_weights(os.path.dirname(lora_path), weight_name=lora_model_name, adapter_name="xl_more_art-full")
# pipe.set_adapters(["photomaker", "xl_more_art-full"], adapter_weights=[1.0, 0.5])
pipe.fuse_lora()
### define the input ID images
input_folder_name = './examples/newton_man'
image_basename_list = os.listdir(input_folder_name)
image_path_list = sorted([os.path.join(input_folder_name, basename) for basename in image_basename_list])
input_id_images = []
for image_path in image_path_list:
input_id_images.append(load_image(image_path))
# Note that the trigger word `img` must follow the class word for personalization
prompt = "a half-body portrait of a man img wearing the sunglasses in Iron man suit, best quality"
negative_prompt = "(asymmetry, worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), open mouth, grayscale"
num_steps = 50
generator = torch.Generator(device=device).manual_seed(42)
images = pipe(
prompt=prompt,
input_id_images=input_id_images,
negative_prompt=negative_prompt,
num_images_per_prompt=1,
num_inference_steps=num_steps,
start_merge_step=10,
generator=generator,
).images[0]
images.save('out_photomaker.png')
```
推理结果为本篇文章的标题图片，Enjoy!