Feature Request: RPC upstream changes from main lcpp

[Panchovix]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Hello, thanks for all your work as always.

I was wondering if RPC updates and changes could be ported from main lcpp into iklcpp.

Nowadays you can get pretty good performance, even when offloading to RAM and also using RPC.

I have these examples on GLM 4.6 full on VRAM on lcpp: ggml-org/llama.cpp#16625 (reply in thread)

I also tried i.e. DeepSeek R1 0528 Q3_K_XL offloading about 25 layers to CPU, 5 layers to RPC (CUDA) and 30 layers to the main PC (CUDA) and got just about 10-20% perf penalty.

Motivation

This would let people use multiple PCs to offload more into devices like CUDA, that in example, on my case, doesn't fit anymore on another system.

Speed penalty is pretty low so it would be worth to consider maybe.

Possible Implementation

Not sure exactly besides porting RPC stuff into here.

As @ubergarm mentioned here https://huggingface.co/ubergarm/Kimi-K2-Thinking-GGUF/discussions/13#691bb8df9287514645b7cc35, it seems to start on ggml-org/llama.cpp#16276 commit.

[firecoperana]

RPC already works here, but for each GPU in remote machine, you need to start a RPC server. In mainline's test, there is not much improvement in token generation speed, but prompt processing has large performance jump. If you only care about token generation speed, it's already usable.

[ikawrakow]

RPC is not something I have looked into. What are we missing?

But let's assume the missing mainline pieces are brought over. Are there any benefits from ik_llama.cpp over llama.cpp with that? My concept is that with the extra latency added for the RPC calls there will barely be any difference between the two.

[Panchovix]

@firecoperana I would be really interested in the improvement of prompt processing speed.

@ikawrakow on iklcpp when offloading to CPU is faster for me on PP t/s vs lcpp. And when testing that case on lcpp (rpc + offloading to CPU), speeds are almost identical to use the GPU on the same system.

[firecoperana]

I have thought about porting them here, but it won't be my top priority. I didn't see many RPC users here and there are other things that could be improved here that could benefit more users, not to mention the complexity to do the port.

In your case, can you compare the speed difference before and after mainline's PR since you only have 1 GPU in rpc machine and I wonder what improvement will that be?

[ikawrakow]

@Panchovix

Would you mind posting some benchmarks with your RPC setup with mainline llama.cpp and with ik_llama.cpp? Thanks.

[Panchovix]

Sure! Once I get home I will post some results.

[Panchovix]

Just an small question at how to run it.

On normal lcpp, I set on my Windows PC with a RTX 5090:

.\rpc-server.exe -H 192.168.50.2 -p 50052

My Linux PC has: 5090x2, 4090x2, A6000, A40 and 3090. But for this test I did export CUDA_VISIBLE_DEVICES=2,0,1,4,5 which is: 5090, 4090, 4090, A40, A6000.

So I run it with:

./llama-server \
  -m '/run/media/pancho/MyDrive/models_llm_2tb/GLM-4.5-IQ4_XS.gguf' \
  -c 32768 --rpc 192.168.50.2:50052 \
  --no-mmap \
  -ngl 999 \
  -ot "blk.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16).ffn.=CUDA0" \
  -ot "blk.(17|18|19|20|21|22|23|24|25|26).ffn.=CUDA1" \
  -ot "blk.(27|28|29|30|31|32|33|34|35|36).ffn.=CUDA2" \
  -ot "blk.(37|38|39|40|41|42|43|44|45|46|47|48|49|50).ffn.=RPC0[192.168.50.2:50052]" \
  -ot "blk.(51|52|53|54|55|56|57|58|59|60).ffn.=CUDA3" \
  -ot "blk.(61|62|63|64|65|66|67|68|69|70).ffn.=CUDA3" \
  -ot "blk.(71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92).ffn.=CUDA4" \
  -mg 0 \
  -ub 1792 --device CUDA0,CUDA1,CUDA2,RPC0,CUDA3,CUDA4

How would the equivalent commands look for iklcpp?

I.e. I get these speeds using that:

prompt eval time =    5612.13 ms /  4453 tokens (    1.26 ms per token,   793.46 tokens per second)
       eval time =   20509.94 ms /   493 tokens (   41.60 ms per token,    24.04 tokens per second)

While running on the PC natively with the other 5090, I get 866.15 t/s PP, 27.93 t/s TG. So there's a hit but those speeds are pretty acceptable for RPC.

Note: It is really important to be able to move the RPC0 device to not the end, else speed tanks to about 200 t/s PP and 10 t/s TG.

[firecoperana]

It should be the same mostly, but in --device and -ot, change them to be RPC[192.168.50.2:50052]

[Panchovix]

Okay just tested. Took a bit more since ik lcpp took about 1 hour to build on my windows PC (7800X3D, using -j 5), on Linux it took like 10 mins on a 9900X, with -j 11.

I don't know if I mentioned before, but the NIC is running at X2 3.0, so about 12-13Gbps max.

Built both with

cmake -B build \
    -DGGML_CUDA=ON \
    -DGGML_CUDA_FA_ALL_QUANTS=ON \
    -DGGML_BLAS=OFF \
    -DCMAKE_CUDA_ARCHITECTURES="86;89;120" \
    -DGGML_IQK_FA_ALL_QUANTS=1 \
    -DGGML_SCHED_MAX_COPIES=1 \
    -DGGML_MAX_CONTEXTS=2048 \
    -DGGML_RPC=ON \

Then on Windows:

S H:\ik_llama.cpp\windows\bin\Release> .\rpc-server.exe -H 192.168.50.2 -p 50052

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
WARNING: Host ('192.168.50.2') is != '127.0.0.1'
         Never expose the RPC server to an open network!
         This is an experimental feature and is not secure!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

create_backend: using CUDA backend
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
Starting RPC server v2.0.1
  endpoint       : 192.168.50.2:50052
  local cache    : n/a
  backend memory : 30841 MB

Then on Linux:

./llama-server \
  -m '/run/media/pancho/MyDrive/models_llm_2tb/GLM-4.5-IQ4_XS.gguf' \
  -c 32768 --rpc 192.168.50.2:50052 \
  --no-mmap \
  -ngl 999 \
  -ot "blk.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16).ffn.=CUDA0" \
  -ot "blk.(17|18|19|20|21|22|23|24|25|26).ffn.=CUDA1" \
  -ot "blk.(27|28|29|30|31|32|33|34|35|36).ffn.=CUDA2" \
  -ot "blk.(37|38|39|40|41|42|43|44|45|46|47|48|49|50).ffn.=RPC[192.168.50.2:50052]" \
  -ot "blk.(51|52|53|54|55|56|57|58|59|60).ffn.=CUDA3" \
  -ot "blk.(61|62|63|64|65|66|67|68|69|70).ffn.=CUDA3" \
  -ot "blk.(71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92).ffn.=CUDA4" \
  -mg 0 \
  -ub 1792 --device CUDA0,CUDA1,CUDA2,RPC[192.168.50.2:50052],CUDA3,CUDA4

Also with this VRAM load:

llm_load_tensors: RPC[192.168.50.2:50052] buffer size = 28432.41 MiB
llm_load_tensors:  CUDA_Host buffer size =   416.25 MiB
llm_load_tensors:      CUDA0 buffer size = 28322.05 MiB
llm_load_tensors:      CUDA1 buffer size = 20045.11 MiB
llm_load_tensors:      CUDA2 buffer size = 19975.17 MiB
llm_load_tensors:      CUDA3 buffer size = 40090.22 MiB
llm_load_tensors:      CUDA4 buffer size = 42017.80 MiB

First got this error

INFO [                    main] HTTP server listening | tid="140442289651712" timestamp=1763595430 n_threads_http="23" port="8080" hostname="127.0.0.1"
INFO [            update_slots] all slots are idle | tid="140442289651712" timestamp=1763595430
INFO [      log_server_request] request | tid="140429148282880" timestamp=1763595442 remote_addr="127.0.0.1" remote_port=59388 status=200 method="GET" path="/v1/models" params={}
INFO [      log_server_request] request | tid="140429148282880" timestamp=1763595442 remote_addr="127.0.0.1" remote_port=59388 status=200 method="GET" path="/props" params={}
prompt cache: cache size: 0, n_keep: 0, n_discarded: 0, cache_ram_n_min: 0, f_keep: 0.00, cache_ram_similarity: 0.50
INFO [   launch_slot_with_task] slot is processing task | tid="140442289651712" timestamp=1763595445 id_slot=0 id_task=0
INFO [            update_slots] kv cache rm [p0, end) | tid="140442289651712" timestamp=1763595445 id_slot=0 id_task=0 p0=0
CUDA error: out of memory
  current device: 0, in function alloc at /run/media/pancho/MX500/ChatIAs/ik_llama.cpp/ggml/src/ggml-cuda.cu:397
  cuMemCreate(&handle, reserve_size, &prop, 0)
/run/media/pancho/MX500/ChatIAs/ik_llama.cpp/ggml/src/ggml-cuda.cu:123: CUDA error

Note this doesn't happen on main lcpp.

Then loaded with

./llama-server   -m '/run/media/pancho/MyDrive
/models_llm_2tb/GLM-4.5-IQ4_XS.gguf'   -c 32768 --rpc 192.168.50.2:50052   --no-mmap   -ngl 999   -ot "blk.(0|1|2|3|
4|5|6|7|8|9|10|11|12|13|14|15).ffn.=CUDA0"   -ot "blk.(16|17|18|19|20|21|22|23|24|25).ffn.=CUDA1"   -ot "blk.(26|27|
28|29|30|31|32|33|34|35).ffn.=CUDA2"   -ot "blk.(36|37|38|39|40|41|42|43|44|45|46|47|48|49).ffn.=RPC[192.168.50.2:50
052]"   -ot "blk.(50|51|52|53|54|55|56|57|58|59|60).ffn.=CUDA3"   -ot "blk.(61|62|63|64|65|66|67|68|69|70).ffn.=CUDA3"   -ot "blk.(71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92).ffn.=CUDA4"   -mg 0   -ub 1792 --device CUDA0,CUDA1,CUDA2,RPC[192.168.50.2:50052],CUDA3,CUDA4

(Basically removed 1 layer from CUDA 0 and added one to CUDA 3)

Which gave me this loading VRAM

llm_load_tensors: RPC[192.168.50.2:50052] buffer size = 28432.41 MiB
llm_load_tensors:  CUDA_Host buffer size =   416.25 MiB
llm_load_tensors:      CUDA0 buffer size = 26394.48 MiB
llm_load_tensors:      CUDA1 buffer size = 20045.11 MiB
llm_load_tensors:      CUDA2 buffer size = 19975.17 MiB
llm_load_tensors:      CUDA3 buffer size = 42017.80 MiB
llm_load_tensors:      CUDA4 buffer size = 42017.80 MiB

Then got OOM on RPC when starting to generate, so reduced the context to 8K (-c 8192) to fit it correctly.

Finally here it loaded fine without OOM, but I noticed that on first gen sometimes (via Sillytavern) it doesn't output, but it does on 2nd gen.

After some tries managed to make it work first try, and got these speeds:

INFO [           print_timings] prompt eval time     =   19910.28 ms /  4453 tokens (    4.47 ms per token,   223.65 tokens per second) | tid="140631478865920" timestamp=1763596848 id_slot=0 id_task=0 t_prompt_processing=19910.277 n_prompt_tokens_processed=4453 t_token=4.471205254884347 n_tokens_second=223.65334244219707
INFO [           print_timings] generation eval time =   27332.64 ms /   444 runs   (   61.56 ms per token,    16.24 tokens per second) | tid="140631478865920" timestamp=1763596848 id_slot=0 id_task=0 t_token_generation=27332.636 n_decoded=444 t_token=61.55999099099099 n_tokens_second=16.244316867205928
INFO [           print_timings]           total time =   47242.91 ms | tid="140631478865920" timestamp=1763596848 id_slot=0 id_task=0 t_prompt_processing=19910.277 t_token_generation=27332.636 t_total=47242.913

And here's the complete log if it helps (on pastebin) https://pastebin.com/vr1tswFv

If any other test is needed just let me know.

[ikawrakow]

@Panchovix Thanks!

What happens if you don't use RPC and just put the RPC layers on the local CPU? Or better, just the MoE tensors in these layers.

The reason I'm asking is that 223 t/s PP seems really low. I get 230 t/s on 2x3090 with all MoE tensors in RAM.

Another comment is that -ub 1792 is a very strange u-batch size and likely results in a non-negligible PP speed drop. You are not setting the batch size, which is 2048 by default, which means that a batch will get processed as one u-batch of 1792 and a second u-batch of 256, which is bad for a MoE model. I don't know how llama.cpp behaves -- perhaps they set batch size automatically to 1792 fort you? -- but ik_llama.cpp doesn't do that, so you need -b 1792 -ub 1792 if 1792 is the largest u-batch you can have.

[firecoperana]

From personal experience, I usually put last n layers to RPC, not in the middle of the device. You can also compare vram usage of each device with mainline. They should be the same. Also I prefer to specify tensor split, so it will something like --tensor-split 999,0,0,0,0,0

[ikawrakow]

@firecoperana

@Panchovix went through this journey with mainline llama.cpp:

ggml-org/llama.cpp#16625 (reply in thread)

His PP increased from 189 t/s (default device order) to 263 t/s (putting RPC at the end) to 680 t/s (putting RPC in the middle). Clearly behavior there is very different from behavior here (and this is an actual issue when people come here and try to do the exact same thing that they did with llama.cpp).

In any case, I went through the trouble of making a GLM-4.6 quant that is more inline in terms of size with the IQ4_XS model he is using. Here is the sweep bench I get with that with ik_llama.cpp, offloading just 6 out of 92 layers to VRAM in a 2x3090 box:

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
4096	64	0	13.843	295.90	5.730	11.17
4096	64	4096	14.808	276.60	6.035	10.61
4096	64	8192	16.011	255.82	6.115	10.47
4096	64	12288	17.197	238.19	6.506	9.84
4096	64	16384	18.406	222.53	6.805	9.41
4096	64	20480	19.667	208.26	7.070	9.05
4096	64	24576	20.749	197.41	7.373	8.68
4096	64	28672	22.151	184.92	7.765	8.24

TG is basically running CPU-only (Ryzen-3995WX, released in 2020), so being just a factor of 2 lower than full GPU offload is not too bad.

[Panchovix]

@ikawrakow Okay tested first with only those layers on CPU (9900X with 192GB 6000Mhz RAM), so

./llama-server \
  -m '/run/media/pancho/MyDrive/models_llm_2tb/GLM-4.5-IQ4_XS.gguf' \
  -c 32768 \
  --no-mmap \
  -ngl 999 \
  -ot "blk.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15).ffn.=CUDA0" \
  -ot "blk.(16|17|18|19|20|21|22|23|24|25).ffn.=CUDA1" \
  -ot "blk.(26|27|28|29|30|31|32|33|34|35).ffn.=CUDA2" \
  -ot "blk.(36|37|38|39|40|41|42|43|44|45|46|47|48|49).ffn.=CPU" \
  -ot "blk.(50|51|52|53|54|55|56|57|58|59|60).ffn.=CUDA3" \
  -ot "blk.(61|62|63|64|65|66|67|68|69|70).ffn.=CUDA3" \
  -ot "blk.(71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86|87|88|89|90|91|92).ffn.=CUDA4" \
  -mg 0 \
  -ub 1792

llm_load_tensors:        CPU buffer size = 26986.09 MiB
llm_load_tensors:  CUDA_Host buffer size =   416.25 MiB
llm_load_tensors:      CUDA0 buffer size = 26534.35 MiB
llm_load_tensors:      CUDA1 buffer size = 20184.99 MiB
llm_load_tensors:      CUDA2 buffer size = 20184.99 MiB
llm_load_tensors:      CUDA3 buffer size = 42297.56 MiB
llm_load_tensors:      CUDA4 buffer size = 42694.79 MiB

Which is noticeably faster on PP, a bit faster on TG:

INFO [           print_timings] prompt eval time     =   10496.36 ms /  4453 tokens (    2.36 ms per token,   424.24 tokens per second) | tid="139823625699328" timestamp=1763645562 id_slot=0 id_task=0 t_prompt_processing=10496.358 n_prompt_tokens_processed=4453 t_token=2.3571430496294634 n_tokens_second=424.2423895983731
INFO [           print_timings] generation eval time =   29693.92 ms /   482 runs   (   61.61 ms per token,    16.23 tokens per second) | tid="139823625699328" timestamp=1763645562 id_slot=0 id_task=0 t_token_generation=29693.922 n_decoded=482 t_token=61.605647302904565 n_tokens_second=16.232278107284046
INFO [           print_timings]           total time =   40190.28 ms | tid="139823625699328" timestamp=1763645562 id_slot=0 id_task=0 t_prompt_processing=10496.358 t_token_generation=29693.922 t_total=40190.28

Also tried -ub = b (1792), and got

INFO [           print_timings] prompt eval time     =    8814.15 ms /  4453 tokens (    1.98 ms per token,   505.21 tokens per second) | tid="140653791744000" timestamp=1763645866 id_slot=0 id_task=0 t_prompt_processing=8814.146 n_prompt_tokens_processed=4453 t_token=1.9793725578261847 n_tokens_second=505.210601231248
INFO [           print_timings] generation eval time =   23171.18 ms /   376 runs   (   61.63 ms per token,    16.23 tokens per second) | tid="140653791744000" timestamp=1763645866 id_slot=0 id_task=0 t_token_generation=23171.181 n_decoded=376 t_token=61.62548138297873 n_tokens_second=16.227053769939477
INFO [           print_timings]           total time =   31985.33 ms | tid="140653791744000" timestamp=1763645866 id_slot=0 id_task=0 t_prompt_processing=8814.146 t_token_generation=23171.181 t_total=31985.327

So as you say, b = ub below 2048 is faster. I was using 1792 since it was the max ubatch size for my VRAM. It is factible that lcpp defaults batch size to also 1792 then.

Then also tried 2x5090 (each at X8 5.0) and the rest on CPU with (had to remove a lot of layers on CUDA 0, since offloading uses more VRAM IIRC on compute buffers):

  ./llama-server \
  -m '/run/media/pancho/MyDrive/models_llm_2tb/GLM-4.5-IQ4_XS.gguf' \
  -c 32768 \
  --no-mmap \
  -ngl 999 \
  -ot "blk.(0|1|2|3|4|5|6|7|8|9|10|11).ffn.=CUDA0" \
  -ot "blk.(12|13|14|15|16|17|18|19|20|21).ffn.=CUDA1" \
  -ot "exps=CPU" \
  -mg 0 \
  -ub 1792 -b 1792

llm_load_tensors:        CPU buffer size = 133875.00 MiB
llm_load_tensors:  CUDA_Host buffer size =   416.25 MiB
llm_load_tensors:      CUDA0 buffer size = 21299.17 MiB
llm_load_tensors:      CUDA1 buffer size = 23708.61 MiB

and got (with b and ub 1792)

INFO [           print_timings] prompt eval time     =   18341.68 ms /  4453 tokens (    4.12 ms per token,   242.78 tokens per second) | tid="140338398220288" timestamp=1763647439 id_slot=0 id_task=0 t_prompt_processing=18341.682 n_prompt_tokens_processed=4453 t_token=4.118949472265888 n_tokens_second=242.78035133310019
INFO [           print_timings] generation eval time =   86223.37 ms /   720 runs   (  119.75 ms per token,     8.35 tokens per second) | tid="140338398220288" timestamp=1763647439 id_slot=0 id_task=0 t_token_generation=86223.372 n_decoded=720 t_token=119.75468333333333 n_tokens_second=8.350404110848274

Finally tested RPC again but with b = ub = 1792, and got this speed:

INFO [           print_timings] prompt eval time     =   27668.75 ms /  4453 tokens (    6.21 ms per token,   160.94 tokens per second) | tid="139830824103936" timestamp=1763647743 id_slot=0 id_task=0 t_prompt_processing=27668.754 n_prompt_tokens_processed=4453 t_token=6.213508645856726 n_tokens_second=160.9396650098519
INFO [           print_timings] generation eval time =   26488.21 ms /   389 runs   (   68.09 ms per token,    14.69 tokens per second) | tid="139830824103936" timestamp=1763647743 id_slot=0 id_task=0 t_token_generation=26488.206 n_decoded=389 t_token=68.09307455012853 n_tokens_second=14.685781286962206
INFO [           print_timings]           total time =   54156.96 ms | tid="139830824103936" timestamp=1763647743 id_slot=0 id_task=0 t_prompt_processing=27668.754 t_token_generation=26488.206 t_total=54156.96

(Not sure why is slower?)

@firecoperana If I use RPC at the end my speed really, really tanks for some reason. I guess it depends if using -ts or -ot. Since I have GPUs with different sizes I can't quite use -ts correctly and -ot works perfectly. VRAM usage seems to diverge a little mostly on CUDA 0.

But it seems, even when using --device, by default it puts RPC as first device instead of CUDA0 itself, so I think that may be the reason of slower performance (basically using X2 3.0 for the NIC instead of X8 5.0 of the real CUDA 0 on the main system)

[ikawrakow]

@Panchovix

So, GLM-4.5/4.6 has shared experts. You definitely do not want to have those on the CPU. Neither do you want to have small normalization tensors (exp_probs_b.bias, ffn_gate_inp.weight) left on the CPU. So, your CPU overrides need to be

-ot "blk\.(36|37|38|39|40|41|42|43|44|45|46|47|48|49)\.ffn_(up|gate|down)_exps\.weight=CPU"

The other thing is that you are sprinkling attention and ffn tensors all over the place with your overrides. This results in many graph splits that slow things down. Ideally you want attention, MoE, and normalization tensors of the same layer to end up on the same GPU. My guess is that with more careful overrides you can get another 10-20% improvement of PP and TG in the 5xGPU+CPU case.

Which means that having a better RPC will be a 10-20% gain for one user who prefers to buy 6 relatively expensive GPUs, and because there aren't enough slots in the GPU box, wants to use RPC to utilize the 6th GPU in another box. Instead of, for instance, investing the money for the 5th and 6th GPU into a better CPU and more RAM, and most likely ending up with a better TG performance. 😄 And, after having spent the time to do it, there will be not that much of a difference to mainline llama.cpp, which you can already successfully use.