Inference with Starcoder model finetuned by lora

[edwardelric1202]

Hi, Can you give some advice about how to inference finetuned Starcoder model with this code? Since lora finetune changed some of layers of the model, some of the code in starcoder.cpp should be changed, how can I use this code to inference with my finetuned Starcoder model?

[NouamaneTazi]

Unfortunately this is not supported yet, but I think llama.cpp does support it.
It would be great if somebody can adapt that to starcoder.cpp
ggml-org/llama.cpp#820