Enable CUDA graphs for MoE models + GPT-OSS support#689

Merged

ikawrakow merged 31 commits intomainfrom

ik/try_cuda_graphs

Aug 15, 2025

Commits on Aug 9, 2025

gmp-oss: common
Iwan Kawrakow
committed
gpt-oss: attnetion sinks, swiglu_oai
Iwan Kawrakow
committed

Commits on Aug 10, 2025

gpt-oss: WIP llama
Iwan Kawrakow
committed
gpt-oss: CPU seems to be working
Iwan Kawrakow
committed
CUDA: ADD_ID
Iwan Kawrakow
committed
gpt-oss: Seems to be working on CUDA
Iwan Kawrakow
committed
gpt-oss: add sinks to the attn-vec kernels
Iwan Kawrakow
committed

Commits on Aug 11, 2025

CUDA: add head size of 64 to new mma
Iwan Kawrakow
committed

Commits on Aug 12, 2025

gpt-oss: add ability to use -fmoe (only CUDA for now)
Iwan Kawrakow
committed
Move row sums to the write place
Iwan Kawrakow
committed
Add sinks to iqk flash attention
Iwan Kawrakow
committed
gpt_oss: Implement -fmoe on the CPU
Iwan Kawrakow
committed
Simdify swiglu_oai
Iwan Kawrakow
committed

Commits on Aug 13, 2025

llama: factor out model loader
Iwan Kawrakow
committed
Builds successfully
Iwan Kawrakow
committed
It runs, but mmap does not work
Iwan Kawrakow
committed
Fix llama_mmap so mmap works
Iwan Kawrakow
committed
Minor
Iwan Kawrakow
committed
Fix CUDA after latest changes
Iwan Kawrakow
committed

Commits on Aug 14, 2025

Attempt to use CUDA graphs with MoE models - not working
Iwan Kawrakow
committed
CUDA graphs WIP - still not working
Iwan Kawrakow
committed
CUDA graphs - seems to be working
Iwan Kawrakow
committed
Make q8_0 cache work for DeepSeek models with CUDA graphs
Iwan Kawrakow
committed
cuda: cpy for q6_0
Iwan Kawrakow
committed
Fix llama_mmap on non-Linux platforms
Iwan Kawrakow
committed
Adding forgotten file
Iwan Kawrakow
committed
Iterating on Windows build failures
Iwan Kawrakow
committed
cuda: re-add q8_0 -> q8_0 transpose
Iwan Kawrakow
committed

Commits on Aug 15, 2025

Disable graphs without -fmoe
Iwan Kawrakow
committed
Minor
Iwan Kawrakow
committed
Turn graphs on by default
Iwan Kawrakow
committed