[Bug] llama-factory训练报错：ModuleNotFoundError: No module named 'cpuinfer_ext.sft_moe'; 'cpuinfer_ext' is not a package #1567
Description
检查清单
- 1. 我已经搜索过相关问题，但未能获得预期的帮助
- 2. 该问题在最新版本中尚未修复
- 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
- 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
- 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭
问题描述
https://github.com/hiyouga/LLaMA-Factory/blob/main/examples/train_lora/deepseek2_lora_sft_kt.yaml
用这个脚本训练报错，如下：
Injecting model.layers.1.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.1.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_layernorm as default
Injecting model.layers.1.self_attn.kv_b_proj as default
Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbedding
Traceback (most recent call last):
File "/mnt/workspace/miniconda3/envs/LF_KT/bin/llamafactory-cli", line 8, in <module>
sys.exit(main())
^^^^^^
File "/mnt/workspace/LLaMA-Factory/src/llamafactory/cli.py", line 24, in main
launcher.launch()
File "/mnt/workspace/LLaMA-Factory/src/llamafactory/launcher.py", line 157, in launch
run_exp()
File "/mnt/workspace/LLaMA-Factory/src/llamafactory/train/tuner.py", line 132, in run_exp
_training_function(config={"args": args, "callbacks": callbacks})
File "/mnt/workspace/LLaMA-Factory/src/llamafactory/train/tuner.py", line 91, in _training_function
run_sft_kt(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/mnt/workspace/LLaMA-Factory/src/llamafactory/train/ksft/workflow.py", line 47, in run_sft
model = load_model(tokenizer, model_args, finetuning_args, training_args.do_train)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/workspace/LLaMA-Factory/src/llamafactory/model/loader.py", line 151, in load_model
model = load_kt_pretrained_model(config, model_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/workspace/LLaMA-Factory/src/llamafactory/model/model_utils/ktransformers.py", line 110, in load_kt_pretrained_model
optimize_and_load_gguf(model, optimize_config_path, gguf_path, config)
File "/mnt/workspace/miniconda3/envs/LF_KT/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf
inject(module, optimize_config, model_config, weights_loader)
File "/mnt/workspace/miniconda3/envs/LF_KT/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 42, in inject
inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
File "/mnt/workspace/miniconda3/envs/LF_KT/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 42, in inject
inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
File "/mnt/workspace/miniconda3/envs/LF_KT/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 42, in inject
inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
File "/mnt/workspace/miniconda3/envs/LF_KT/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 31, in inject
module_cls=getattr(__import__(import_module_name, fromlist=[""]), import_class_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/workspace/miniconda3/envs/LF_KT/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 32, in <module>
from cpuinfer_ext.sft_moe import SFT_MOEConfig, SFT_MOE
ModuleNotFoundError: No module named 'cpuinfer_ext.sft_moe'; 'cpuinfer_ext' is not a package
复现步骤
训练命令：
llamafactory-cli train examples/train_lora/deepseek2_lora_sft_kt.yaml环境信息
环境：
llamafactory 0.9.4.dev0
ktransformers 0.3.2+cu121torch27fancy
机器不支持AMX