Skip to content

沐曦曦云 C500 支持 #4768

@Chenhb123

Description

@Chenhb123

Checklist:

描述问题

您好,我注意到paddlepaddle已经支持沐曦曦云 C500 ,但是paddlex尚未提供支持,跟您确认近期是否有沐曦曦云 C500 适配计划,期待您的回复。

复现

使用#4453的提交手动添加沐曦卡支持后,进行OCR产线测试报错,看日志可能是由于无法正确处理 bfloat16 类型的转换指令,但是日志最后也有解析内容输出:

paddlex --pipeline /app/llmdoc/yaml/gpu/OCR.yaml         --input general_ocr_002.png         --use_doc_orientation_classify False         --use_doc_unwarping False         --use_textline_orientation False         --save_path ./output         --device metax_gpu:0
Creating model: ('PP-OCRv5_server_det', None)
Model files already exist. Using cached files. To redownload, please delete the directory manually: `/app/llmdoc/models/official_models/PP-OCRv5_server_det`.
/opt/conda/lib/python3.10/site-packages/paddle/utils/cpp_extension/extension_utils.py:703: UserWarning: No ccache found. Please be aware that recompiling all source files may be required. You can download and install ccache from: https://github.com/ccache/ccache/blob/master/doc/INSTALL.md
  warnings.warn(warning_message)
I1127 10:01:14.253327 39304 init.cc:238] ENV [CUSTOM_DEVICE_ROOT]=/opt/conda/lib/python3.10/site-packages/paddle_custom_device
I1127 10:01:14.253371 39304 init.cc:146] Try loading custom device libs from: [/opt/conda/lib/python3.10/site-packages/paddle_custom_device]
I1127 10:01:23.826114 39304 custom_device_load.cc:51] Succeed in loading custom runtime in lib: /opt/conda/lib/python3.10/site-packages/paddle_custom_device/libpaddle-metax-gpu.so
I1127 10:01:23.826201 39304 custom_device_load.cc:58] Skipped lib [/opt/conda/lib/python3.10/site-packages/paddle_custom_device/libpaddle-metax-gpu.so]: no custom engine Plugin symbol in this lib.
I1127 10:01:23.843274 39304 custom_kernel.cc:68] Succeed in loading 970 custom kernel(s) from loaded lib(s), will be used like native ones.
I1127 10:01:23.843935 39304 init.cc:158] Finished in LoadCustomDevice with libs_path: [/opt/conda/lib/python3.10/site-packages/paddle_custom_device]
I1127 10:01:23.844010 39304 init.cc:244] CustomDevice: metax_gpu, visible devices count: 1
Creating model: ('PP-OCRv5_server_rec', None)
Model files already exist. Using cached files. To redownload, please delete the directory manually: `/app/llmdoc/models/official_models/PP-OCRv5_server_rec`.
[10:01:34.931][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
 
[10:01:34.932][MCC][E]StatRecompileKernel recompile fail
[10:01:34.932][MCR][E]mc_recompile.cpp         :383 : Recompile kernel failed
[10:01:34.932][MCR][E]mc_runtime_api.cpp       :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:35.767][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
 
[10:01:35.768][MCC][E]StatRecompileKernel recompile fail
[10:01:35.768][MCR][E]mc_recompile.cpp         :383 : Recompile kernel failed
[10:01:35.768][MCR][E]mc_runtime_api.cpp       :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:35.945][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
 
[10:01:35.946][MCC][E]StatRecompileKernel recompile fail
[10:01:35.946][MCR][E]mc_recompile.cpp         :383 : Recompile kernel failed
[10:01:35.946][MCR][E]mc_runtime_api.cpp       :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:36.733][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
 
[10:01:36.733][MCC][E]StatRecompileKernel recompile fail
[10:01:36.733][MCR][E]mc_recompile.cpp         :383 : Recompile kernel failed
[10:01:36.733][MCR][E]mc_runtime_api.cpp       :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:36.907][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
 
[10:01:36.907][MCC][E]StatRecompileKernel recompile fail
[10:01:36.907][MCR][E]mc_recompile.cpp         :383 : Recompile kernel failed
[10:01:36.907][MCR][E]mc_runtime_api.cpp       :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:37.103][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
 
[10:01:37.103][MCC][E]StatRecompileKernel recompile fail
[10:01:37.103][MCR][E]mc_recompile.cpp         :383 : Recompile kernel failed
[10:01:37.103][MCR][E]mc_runtime_api.cpp       :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:37.276][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
 
[10:01:37.276][MCC][E]StatRecompileKernel recompile fail
[10:01:37.276][MCR][E]mc_recompile.cpp         :383 : Recompile kernel failed
[10:01:37.276][MCR][E]mc_runtime_api.cpp       :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:37.472][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
 
[10:01:37.473][MCC][E]StatRecompileKernel recompile fail
[10:01:37.473][MCR][E]mc_recompile.cpp         :383 : Recompile kernel failed
[10:01:37.473][MCR][E]mc_runtime_api.cpp       :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
[10:01:37.657][MCC][E]online build program ret(17920) error: Intrinsic has incorrect return type!
i16 (float)* @llvm.mxc.cvt.f32tobf16
in function _ZN3phi5funcs29KernelMaxPool2DGradCompatibleINS_5dtype8bfloat16EiEEvPKT_S6_S6_T0_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_S7_PS4_NS0_20FastDivModForPoolingIS7_EEb
fatal error: error in backend: Broken function found, compilation aborted!
mxcc: error: clang frontend command failed with exit code 70 (use -v to see invocation)
mxcc version 1.0.0 (163758af6a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/maca/mxgpu_llvm/bin
mxcc: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
 
[10:01:37.657][MCC][E]StatRecompileKernel recompile fail
[10:01:37.657][MCR][E]mc_recompile.cpp         :383 : Recompile kernel failed
[10:01:37.657][MCR][E]mc_runtime_api.cpp       :341 : 39304: [7fa8ec124740] mcLaunchKernel: Returned mcErrorRecompile
{'res': {'input_path': 'general_ocr_002.png', 'page_index': None, 'model_settings': {'use_doc_preprocessor': False, 'use_textline_orientation': False}, 'dt_polys': array([[[220,   0],
        ...,
        [220,  79]],

       ...,

       [[ 97, 458],
        ...,
        [ 97, 479]]], dtype=int16), 'text_det_params': {'limit_side_len': 736, 'limit_type': 'min', 'thresh': 0.3, 'max_side_limit': 4000, 'box_thresh': 0.6, 'unclip_ratio': 1.5}, 'text_type': 'general', 'textline_orientation_angles': array([-1, ..., -1]), 'text_rec_score_thresh': 0.0, 'return_word_box': False, 'rec_texts': ['', '', '', 'a', '', '', '', '', '', '', '1', '', '', '', '', 'C', '', '', '', '1', '', '', '', '', '', '', ''], 'rec_scores': array([0., ..., 0.]), 'rec_polys': array([[[220,   0],
        ...,
        [220,  79]],

       ...,

       [[ 97, 458],
        ...,
        [ 97, 479]]], dtype=int16), 'rec_boxes': array([[220, ...,  79],
       ...,
       [ 97, ..., 479]], dtype=int16)}}
Segmentation fault (core dumped)
  1. 高性能推理

  2. 服务化部署

    • 您是否完全按照服务化部署文档教程跑通了流程?

    • 您在服务化部署中是否有使用高性能推理插件?

    • 您使用了哪一种服务化部署方案?

    • 如果是多语言调用的问题,请给出调用示例子。

  3. 端侧部署

    • 您是否完全按照端侧部署文档教程跑通了流程?

    • 您使用的端侧设备是?对应的PaddlePaddle版本和PaddleLite版本分别是什么?

  4. 您使用的模型数据集是?

  5. 请提供您出现的报错信息及相关log

环境

  1. 请提供您使用的PaddlePaddle、PaddleX版本号、Python版本号
python -V
Python 3.10.10
pip list installed | grep paddle
paddle-metax-gpu                         3.3.0.dev20251126+maca0.0.0
paddleclas                               2.5.2
paddleocr                                3.3.2
paddlepaddle                             3.3.0.dev20251126
paddlex                                  3.3.10
  1. 请提供您使用的操作系统信息,如Linux/Windows/MacOS

  2. 请问您使用的CUDA/cuDNN的版本号是?

 mx-smi 
mx-smi  version: 2.2.4

=================== MetaX System Management Interface Log ===================
Timestamp                                         : Thu Nov 27 10:04:01 2025

Attached GPUs                                     : 2
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.4                        Kernel Mode Driver Version: 3.3.12          |
| MACA Version: 2.33.1.13             BIOS Version: 1.29.1.0                      |
|------------------------------------+---------------------+----------------------+
| GPU         NAME                   | Bus-id              | GPU-Util             |
| Temp        Pwr:Usage/Cap          | Memory-Usage        | GPU-State            |
|====================================+=====================+======================|
| 0           MetaX C500             | 0000:16:00.0        | 0%                   |
| 40C         38W / 350W             | 858/65536 MiB       | Available            |
+------------------------------------+---------------------+----------------------+
| 1           MetaX C500             | 0000:98:00.0        | 0%                   |
| 39C         37W / 350W             | 858/65536 MiB       | Available            |
+------------------------------------+---------------------+----------------------+

+---------------------------------------------------------------------------------+
| Process:                                                                        |
|  GPU                    PID         Process Name                 GPU Memory     |
|                                                                  Usage(MiB)     |
|=================================================================================|
|  no process found                                                               |
+---------------------------------------------------------------------------------+

End of Log

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions