support dpo orpo and simpo#1631

Draft

poryfly wants to merge 4 commits intokvcache-ai:mainfrom

poryfly:dpo-support

Commits on Nov 20, 2025

support dpo orpo and simpo
unknown
committed
fix bug for KTDpoTrainer
unknown
committed
add trl lib requirement
unknown
committed

Commits on Nov 25, 2025

1、remove invalid loss comput for dpo to save mem;2、add qwen3-235-muti-gpu optimizer rule
unknown
committed