-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Closed
Description
Checklist:
- 查找历史相关issue寻求解答
- 翻阅FAQ
- 翻阅PaddleX 文档
- 确认bug是否在新版本里还未修复
描述问题
您好,我注意到paddlepaddle已经支持沐曦曦云 C500 ,但是paddlex尚未提供支持,跟您确认近期是否有沐曦曦云 C500 适配计划,期待您的回复。
复现
使用#4453的提交手动添加沐曦卡支持后,进行OCR产线测试报错,看日志可能是由于无法正确处理 bfloat16 类型的转换指令,但是日志最后也有解析内容输出:
paddlex --pipeline /app/llmdoc/yaml/gpu/OCR.yaml --input general_ocr_002.png --use_doc_orientation_classify False --use_doc_unwarping False --use_textline_orientation False --save_path ./output --device metax_gpu:0
Creating model: ('PP-OCRv5_server_det', None)
Model files already exist. Using cached files. To redownload, please delete the directory manually: `/app/llmdoc/models/official_models/PP-OCRv5_server_det`.
/opt/conda/lib/python3.10/site-packages/paddle/utils/cpp_extension/extension_utils.py:703: UserWarning: No ccache found. Please be aware that recompiling all source files may be required. You can download and install ccache from: https://github.com/ccache/ccache/blob/master/doc/INSTALL.md
warnings.warn(warning_message)
I1127 10:01:14.253327 39304 init.cc:238] ENV [CUSTOM_DEVICE_ROOT]=/opt/conda/lib/python3.10/site-packages/paddle_custom_device
I1127 10:01:14.253371 39304 init.cc:146] Try loading custom device libs from: [/opt/conda/lib/python3.10/site-packages/paddle_custom_device]
I1127 10:01:23.826114 39304 custom_device_load.cc:51] Succeed in loading custom runtime in lib: /opt/conda/lib/python3.10/site-packages/paddle_custom_device/libpaddle-metax-gpu.so
I1127 10:01:23.826201 39304 custom_device_load.cc:58] Skipped lib [/opt/conda/lib/python3.10/site-packages/paddle_custom_device/libpaddle-metax-gpu.so]: no custom engine Plugin symbol in this lib.
I1127 10:01:23.843274 39304 custom_kernel.cc:68] Succeed in loading 970 custom kernel(s) from loaded lib(s), will be used like native ones.
I1127 10:01:23.843935 39304 init.cc:158] Finished in LoadCustomDevice with libs_path: [/opt/conda/lib/python3.10/site-packages/paddle_custom_device]
I1127 10:01:23.844010 39304 init.cc:244] CustomDevice: metax_gpu, visible devices count: 1
Creating model: ('PP-OCRv5_server_rec', None)
Model files already exist. Using cached files. To redownload, please delete the directory manually: `/app/llmdoc/models/official_models/PP-OCRv5_server_rec`.
[10:01:34.931][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
[10:01:34.932][MCC][E]StatRecompileKernel recompile fail
[10:01:34.932][MCR][E]mc_recompile.cpp :383 : Recompile kernel failed
[10:01:34.932][MCR][E]mc_runtime_api.cpp :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:35.767][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
[10:01:35.768][MCC][E]StatRecompileKernel recompile fail
[10:01:35.768][MCR][E]mc_recompile.cpp :383 : Recompile kernel failed
[10:01:35.768][MCR][E]mc_runtime_api.cpp :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:35.945][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
[10:01:35.946][MCC][E]StatRecompileKernel recompile fail
[10:01:35.946][MCR][E]mc_recompile.cpp :383 : Recompile kernel failed
[10:01:35.946][MCR][E]mc_runtime_api.cpp :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:36.733][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
[10:01:36.733][MCC][E]StatRecompileKernel recompile fail
[10:01:36.733][MCR][E]mc_recompile.cpp :383 : Recompile kernel failed
[10:01:36.733][MCR][E]mc_runtime_api.cpp :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:36.907][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
[10:01:36.907][MCC][E]StatRecompileKernel recompile fail
[10:01:36.907][MCR][E]mc_recompile.cpp :383 : Recompile kernel failed
[10:01:36.907][MCR][E]mc_runtime_api.cpp :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:37.103][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
[10:01:37.103][MCC][E]StatRecompileKernel recompile fail
[10:01:37.103][MCR][E]mc_recompile.cpp :383 : Recompile kernel failed
[10:01:37.103][MCR][E]mc_runtime_api.cpp :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:37.276][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
[10:01:37.276][MCC][E]StatRecompileKernel recompile fail
[10:01:37.276][MCR][E]mc_recompile.cpp :383 : Recompile kernel failed
[10:01:37.276][MCR][E]mc_runtime_api.cpp :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:37.472][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
[10:01:37.473][MCC][E]StatRecompileKernel recompile fail
[10:01:37.473][MCR][E]mc_recompile.cpp :383 : Recompile kernel failed
[10:01:37.473][MCR][E]mc_runtime_api.cpp :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:37.657][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
[10:01:37.657][MCC][E]StatRecompileKernel recompile fail
[10:01:37.657][MCR][E]mc_recompile.cpp :383 : Recompile kernel failed
[10:01:37.657][MCR][E]mc_runtime_api.cpp :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
{'res': {'input_path': 'general_ocr_002.png', 'page_index': None, 'model_settings': {'use_doc_preprocessor': False, 'use_textline_orientation': False}, 'dt_polys': array([[[220, 0],
...,
[220, 79]],
...,
[[ 97, 458],
...,
[ 97, 479]]], dtype=int16), 'text_det_params': {'limit_side_len': 736, 'limit_type': 'min', 'thresh': 0.3, 'max_side_limit': 4000, 'box_thresh': 0.6, 'unclip_ratio': 1.5}, 'text_type': 'general', 'textline_orientation_angles': array([-1, ..., -1]), 'text_rec_score_thresh': 0.0, 'return_word_box': False, 'rec_texts': ['', '', '', 'a', '', '', '', '', '', '', '1', '', '', '', '', 'C', '', '', '', '1', '', '', '', '', '', '', ''], 'rec_scores': array([0., ..., 0.]), 'rec_polys': array([[[220, 0],
...,
[220, 79]],
...,
[[ 97, 458],
...,
[ 97, 479]]], dtype=int16), 'rec_boxes': array([[220, ..., 79],
...,
[ 97, ..., 479]], dtype=int16)}}
Segmentation fault (core dumped)-
高性能推理
- 您是否完全按照高性能推理文档教程跑通了流程?
-
服务化部署
-
您是否完全按照服务化部署文档教程跑通了流程?
-
您在服务化部署中是否有使用高性能推理插件?
-
您使用了哪一种服务化部署方案?
-
如果是多语言调用的问题,请给出调用示例子。
-
-
端侧部署
-
您是否完全按照端侧部署文档教程跑通了流程?
-
您使用的端侧设备是?对应的PaddlePaddle版本和PaddleLite版本分别是什么?
-
-
您使用的模型和数据集是?
-
请提供您出现的报错信息及相关log
环境
- 请提供您使用的PaddlePaddle、PaddleX版本号、Python版本号
python -V
Python 3.10.10
pip list installed | grep paddle
paddle-metax-gpu 3.3.0.dev20251126+maca0.0.0
paddleclas 2.5.2
paddleocr 3.3.2
paddlepaddle 3.3.0.dev20251126
paddlex 3.3.10-
请提供您使用的操作系统信息,如Linux/Windows/MacOS
-
请问您使用的CUDA/cuDNN的版本号是?
mx-smi
mx-smi version: 2.2.4
=================== MetaX System Management Interface Log ===================
Timestamp : Thu Nov 27 10:04:01 2025
Attached GPUs : 2
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.4 Kernel Mode Driver Version: 3.3.12 |
| MACA Version: 2.33.1.13 BIOS Version: 1.29.1.0 |
|------------------------------------+---------------------+----------------------+
| GPU NAME | Bus-id | GPU-Util |
| Temp Pwr:Usage/Cap | Memory-Usage | GPU-State |
|====================================+=====================+======================|
| 0 MetaX C500 | 0000:16:00.0 | 0% |
| 40C 38W / 350W | 858/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 1 MetaX C500 | 0000:98:00.0 | 0% |
| 39C 37W / 350W | 858/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| no process found |
+---------------------------------------------------------------------------------+
End of LogMetadata
Metadata
Assignees
Labels
No labels