返回列表
PhotoMaker X Ascend快速启动
2024-01-24
浏览量:942
# Photomaker PhotoMaker是一个由Tencent分享的StableDiffusion的小玩具,在HuggingFace的官网上也挂了好一阵子,最近有时间在Ascend上做一下相关适配,将过程中碰到的一些问题记录在此。 [HuggingFace链接](https://huggingface.co/TencentARC/PhotoMaker) [Github链接](https://github.com/TencentARC/PhotoMaker) # 推理代码 按照README里面提供的推理代码,我们先初步输出一个推理代码 ``` import torch import torch_npu import os from diffusers.utils import load_image from diffusers import EulerDiscreteScheduler from photomaker.pipeline import PhotoMakerStableDiffusionXLPipeline ### Load base model device = "npu:0" base_model_path = "/home/ma-user/work/aicc/l00564131/models/stable-diffusion-xl-base-1.0" photomaker_path = "/home/ma-user/work/aicc/l00564131/models/TencentARC/PhotoMaker/photomaker-v1.bin" pipe = PhotoMakerStableDiffusionXLPipeline.from_pretrained( base_model_path, # can change to any base model based on SDXL torch_dtype=torch.bfloat16, use_safetensors=True, variant="fp16" ).to(device) ### Load PhotoMaker checkpoint pipe.load_photomaker_adapter( os.path.dirname(photomaker_path), subfolder="", weight_name=os.path.basename(photomaker_path), trigger_word="img" # define the trigger word ) pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config) ### Also can cooperate with other LoRA modules # pipe.load_lora_weights(os.path.dirname(lora_path), weight_name=lora_model_name, adapter_name="xl_more_art-full") # pipe.set_adapters(["photomaker", "xl_more_art-full"], adapter_weights=[1.0, 0.5]) pipe.fuse_lora() ### define the input ID images input_folder_name = './examples/newton_man' image_basename_list = os.listdir(input_folder_name) image_path_list = sorted([os.path.join(input_folder_name, basename) for basename in image_basename_list]) input_id_images = [] for image_path in image_path_list: input_id_images.append(load_image(image_path)) # Note that the trigger word `img` must follow the class word for personalization prompt = "a half-body portrait of a man img wearing the sunglasses in Iron man suit, best quality" negative_prompt = "(asymmetry, worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), open mouth, grayscale" num_steps = 50 generator = torch.Generator(device=device).manual_seed(42) images = pipe( prompt=prompt, input_id_images=input_id_images, negative_prompt=negative_prompt, num_images_per_prompt=1, num_inference_steps=num_steps, start_merge_step=10, generator=generator, ).images[0] gen_images.save('out_photomaker.png') ``` 执行上述代码,会出现以下的报错 ``` Loading PhotoMaker components [1] id_encoder from [/home/ma-user/work/aicc/l00564131/models/TencentARC/PhotoMaker]... Loading PhotoMaker components [2] lora_weights from [/home/ma-user/work/aicc/l00564131/models/TencentARC/PhotoMaker] E39999: Inner Error! E39999 The error from device(chipId:0, dieId:0), serial number is 3, an exception occurred during AICPU execution, stream_id:2, task_id:7572, errcode:21008, msg:inner error.[FUNC:ProcessStarsAicpuErrorInfo][FILE:device_error_proc.cc][LINE:1188] TraceBack (most recent call last): Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:task_info.cc][LINE:1612] AICPU Kernel task happen error, retCode=0x2a.[FUNC:GetError][FILE:stream.cc][LINE:1483] Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=7572, errorCode=2a.[FUNC:PrintAicpuErrorInfo][FILE:task_info.cc][LINE:1457] Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=7572, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1483] rtStreamSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50] synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] Traceback (most recent call last): File "/home/ma-user/work/aicc/l00564131/photomaker/PhotoMaker/gogogo.py", line 52, in images = pipe( File "/home/ma-user/anaconda3/envs/chatglm3-demo/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/ma-user/work/aicc/l00564131/photomaker/PhotoMaker/photomaker/pipeline.py", line 386, in __call__ self.scheduler.set_timesteps(num_inference_steps, device=device) File "/home/ma-user/anaconda3/envs/chatglm3-demo/lib/python3.10/site-packages/diffusers/schedulers/scheduling_euler_discrete.py", line 335, in set_timesteps sigmas = torch.from_numpy(sigmas).to(dtype=torch.float32, device=device) RuntimeError: ACL stream synchronize failed, error code:507018 [W NPUStream.cpp:372] Warning: NPU warning, error code is 507018[Error]: [Error]: The aicpu execution is abnormal. Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue EH9999: Inner Error! rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50] EH9999 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) [W NPUStream.cpp:372] Warning: NPU warning, error code is 507018[Error]: [Error]: The aicpu execution is abnormal. Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue EH9999: Inner Error! rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50] EH9999 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] TraceBack (most recent call last): (function npuSynchronizeDevice) /home/ma-user/anaconda3/envs/chatglm3-demo/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up _warnings.warn(warn_message, ResourceWarning) (chatglm3-demo) [root@6ea3ed6b883e PhotoMaker]# ``` 说实话,这部分报错日志能够体现的内容非常少,所以我们还是需要打开原始的host日志进行分析。我们可以在`/root/ascend/log`下面获取到设备运行时的日志,在此目录下执行`grep -rn ERROR ./`有 ``` (chatglm3-demo) [root@6ea3ed6b883e log]# grep -rn ERROR ./ ./debug/plog/plog-1151919_20240124104533663.log:27:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.034.295 [engine.cc:3775]1151919 ProcLogicCqReport:[FINAL][FINAL]Task run failed, device_id=0, stream_id=2, task_id=7572, sqe_type=1(aicpu), errType=0x1(task exception), sqSwStatus=0x5210 ./debug/plog/plog-1151919_20240124104533663.log:28:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.194 [runtime.cc:4347]1151919 SetWatchDogDevStatus:[FINAL][FINAL]There is errInfo of devId=0, tsId=0 ./debug/plog/plog-1151919_20240124104533663.log:29:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.228 [device_error_proc.cc:1188]1151919 ProcessStarsAicpuErrorInfo:[FINAL][FINAL]report error module_type=0, module_name=E39999 ./debug/plog/plog-1151919_20240124104533663.log:30:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.234 [device_error_proc.cc:1188]1151919 ProcessStarsAicpuErrorInfo:[FINAL][FINAL]The error from device(chipId:0, dieId:0), serial number is 3, an exception occurred during AICPU execution, stream_id:2, task_id:7572, errcode:21008, msg:inner error. ./debug/plog/plog-1151919_20240124104533663.log:31:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.288 [stream.cc:3116]1151919 EnterFailureAbort:[FINAL][FINAL]stream_id=2 enter failure abort. ./debug/plog/plog-1151919_20240124104533663.log:32:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.298 [task_info.cc:1718]1151919 SetStarsResultForDavinciTask:[FINAL][FINAL]AICPU Kernel task happen error, retCode=0x2a. ./debug/plog/plog-1151919_20240124104533663.log:33:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.307 [task_info.cc:1612]1151919 PreCheckTaskErr:[FINAL][FINAL]report error module_type=5, module_name=EZ9999 ./debug/plog/plog-1151919_20240124104533663.log:34:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.312 [task_info.cc:1612]1151919 PreCheckTaskErr:[FINAL][FINAL]Kernel task happen error, retCode=0x2a, [aicpu exception]. ./debug/plog/plog-1151919_20240124104533663.log:35:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.325 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception]. ./debug/plog/plog-1151919_20240124104533663.log:36:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.330 [stream.cc:1483]1151919 GetError:[FINAL][FINAL]report error module_type=0, module_name=E39999 ./debug/plog/plog-1151919_20240124104533663.log:37:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.334 [stream.cc:1483]1151919 GetError:[FINAL][FINAL]AICPU Kernel task happen error, retCode=0x2a. ./debug/plog/plog-1151919_20240124104533663.log:38:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.344 [task_info.cc:1457]1151919 PrintAicpuErrorInfo:[FINAL][FINAL]report error module_type=0, module_name=E39999 ./debug/plog/plog-1151919_20240124104533663.log:39:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.348 [task_info.cc:1457]1151919 PrintAicpuErrorInfo:[FINAL][FINAL]Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=7572, errorCode=2a. ./debug/plog/plog-1151919_20240124104533663.log:40:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.381 [task_info.cc:1467]1151919 PrintAicpuErrorInfo:[FINAL][FINAL]Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=7572, fault op_name= ./debug/plog/plog-1151919_20240124104533663.log:41:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.414 [engine.cc:3646]1151919 StarsResumeRtsq:[FINAL][FINAL]stop scheduling in abort failure mode: stream_id=2, sq_id=2, sq_head=3476, task_id=7572, taskType=1. ./debug/plog/plog-1151919_20240124104533663.log:42:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.429 [stream.cc:2904]1151919 StarsWaitForTask:[FINAL][FINAL]Task Wait: device_id=0, stream_id=2 is Abort, runningState=0 ./debug/plog/plog-1151919_20240124104533663.log:43:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.434 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception]. ./debug/plog/plog-1151919_20240124104533663.log:44:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.438 [stream.cc:1483]1151919 GetError:[FINAL][FINAL]report error module_type=0, module_name=E39999 ./debug/plog/plog-1151919_20240124104533663.log:45:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.442 [stream.cc:1483]1151919 GetError:[FINAL][FINAL]Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=7572, fault op_name= ./debug/plog/plog-1151919_20240124104533663.log:46:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.452 [logger.cc:472]1151919 StreamSynchronize:[FINAL][FINAL]Stream synchronize failed, stream_id=2 ./debug/plog/plog-1151919_20240124104533663.log:47:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.468 [api_c.cc:787]1151919 rtStreamSynchronize:[FINAL][FINAL]ErrCode=507018, desc=[aicpu exception], InnerCode=0x715002a ./debug/plog/plog-1151919_20240124104533663.log:48:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.472 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 ./debug/plog/plog-1151919_20240124104533663.log:49:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.036.479 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]rtStreamSynchronize execute failed, reason=[aicpu exception] ./debug/plog/plog-1151919_20240124104533663.log:50:[ERROR] ASCENDCL(1151919,python):2024-01-24-10:46:23.036.489 [stream.cpp:121]1151919 aclrtSynchronizeStream: [FINAL][FINAL]synchronize stream failed, runtime result = 507018 ./debug/plog/plog-1151919_20240124104533663.log:51:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.404 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception]. ./debug/plog/plog-1151919_20240124104533663.log:52:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.415 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception]. ./debug/plog/plog-1151919_20240124104533663.log:53:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.420 [context.cc:306]1151919 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x715002a. ./debug/plog/plog-1151919_20240124104533663.log:54:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.504 [context.cc:311]1151919 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x715002a. ./debug/plog/plog-1151919_20240124104533663.log:55:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.509 [logger.cc:862]1151919 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. ./debug/plog/plog-1151919_20240124104533663.log:56:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.520 [api_c.cc:1762]1151919 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=507018, desc=[aicpu exception], InnerCode=0x715002a ./debug/plog/plog-1151919_20240124104533663.log:57:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.525 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 ./debug/plog/plog-1151919_20240124104533663.log:58:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.050.530 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[aicpu exception] ./debug/plog/plog-1151919_20240124104533663.log:59:[ERROR] ASCENDCL(1151919,python):2024-01-24-10:46:23.050.556 [device.cpp:285]1151919 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 507018. ./debug/plog/plog-1151919_20240124104533663.log:60:[ERROR] APP(1151919,python):2024-01-24-10:46:23.051.537 [log_inner.cpp:76]1151919 $(notdir $(abspath $<)):THPModule_npu_shutdown:58: "[PTA]:"NPU shutdown synchronize device failed."" ./debug/plog/plog-1151919_20240124104533663.log:61:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.571 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception]. ./debug/plog/plog-1151919_20240124104533663.log:62:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.577 [stream.cc:1480]1151919 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception]. ./debug/plog/plog-1151919_20240124104533663.log:63:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.581 [context.cc:306]1151919 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x715002a. ./debug/plog/plog-1151919_20240124104533663.log:64:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.596 [context.cc:311]1151919 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x715002a. ./debug/plog/plog-1151919_20240124104533663.log:65:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.601 [logger.cc:862]1151919 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. ./debug/plog/plog-1151919_20240124104533663.log:66:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.608 [api_c.cc:1762]1151919 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=507018, desc=[aicpu exception], InnerCode=0x715002a ./debug/plog/plog-1151919_20240124104533663.log:67:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.612 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 ./debug/plog/plog-1151919_20240124104533663.log:68:[ERROR] RUNTIME(1151919,python):2024-01-24-10:46:23.051.617 [error_message_manage.cc:50]1151919 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[aicpu exception] ./debug/plog/plog-1151919_20240124104533663.log:69:[ERROR] ASCENDCL(1151919,python):2024-01-24-10:46:23.051.637 [device.cpp:285]1151919 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 507018. ./debug/device-0/device-1151919_20240124104548927.log:77:[ERROR] AICPU(14937,aicpu_scheduler):2024-01-24-10:46:22.212.021 [masked_scatter.cc:96][AICPU][Compute:96][tid:14948]:[MaskedScatter] Data type of input is not support, input data type is [DT_BFLOAT16]. ./debug/device-0/device-1151919_20240124104548927.log:78:[ERROR] CCECPU(14937,aicpu_scheduler):2024-01-24-10:46:22.212.044 [ae_kernel_lib_aicpu.cc:277][TransformKernelErrorCode][tid:14948][AICPU_PROCESSER] call aicpu api RunCpuKernel in libcpu_kernels.so failed, ret:4294967295. ./debug/device-0/device-1151919_20240124104548927.log:79:[ERROR] CCECPU(14937,aicpu_scheduler):2024-01-24-10:46:22.212.054 [aicpusd_event_process.cpp:1448][ExecuteTsKernelTask][tid:14948] Aicpu engine process failed, result[-1], opName[MaskedScatter]. (chatglm3-demo) [root@6ea3ed6b883e log]# ``` 上面的很多信息并不重要,直接看下面,不难发现在./debug/device-0/日志下面有一条很关键的**Data type of input is not support, input data type is [DT_BFLOAT16].**,我们在原始的推理代码中,按照官方建议设置的数据类型为BF16,那么我们将这部分代码修改为HALF(FP16)即可。 最终使用的推理代码为 ``` import torch import torch_npu import os from diffusers.utils import load_image from diffusers import EulerDiscreteScheduler from photomaker.pipeline import PhotoMakerStableDiffusionXLPipeline ### Load base model device = "npu:0" base_model_path = "/home/ma-user/work/aicc/l00564131/models/stable-diffusion-xl-base-1.0" photomaker_path = "/home/ma-user/work/aicc/l00564131/models/TencentARC/PhotoMaker/photomaker-v1.bin" pipe = PhotoMakerStableDiffusionXLPipeline.from_pretrained( base_model_path, # can change to any base model based on SDXL torch_dtype=torch.half, use_safetensors=True, variant="fp16" ).to(device) ### Load PhotoMaker checkpoint pipe.load_photomaker_adapter( os.path.dirname(photomaker_path), subfolder="", weight_name=os.path.basename(photomaker_path), trigger_word="img" # define the trigger word ) pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config) ### Also can cooperate with other LoRA modules # pipe.load_lora_weights(os.path.dirname(lora_path), weight_name=lora_model_name, adapter_name="xl_more_art-full") # pipe.set_adapters(["photomaker", "xl_more_art-full"], adapter_weights=[1.0, 0.5]) pipe.fuse_lora() ### define the input ID images input_folder_name = './examples/newton_man' image_basename_list = os.listdir(input_folder_name) image_path_list = sorted([os.path.join(input_folder_name, basename) for basename in image_basename_list]) input_id_images = [] for image_path in image_path_list: input_id_images.append(load_image(image_path)) # Note that the trigger word `img` must follow the class word for personalization prompt = "a half-body portrait of a man img wearing the sunglasses in Iron man suit, best quality" negative_prompt = "(asymmetry, worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), open mouth, grayscale" num_steps = 50 generator = torch.Generator(device=device).manual_seed(42) images = pipe( prompt=prompt, input_id_images=input_id_images, negative_prompt=negative_prompt, num_images_per_prompt=1, num_inference_steps=num_steps, start_merge_step=10, generator=generator, ).images[0] images.save('out_photomaker.png') ``` 推理结果为本篇文章的标题图片,Enjoy!