GGUF convert support for Vibevoice

[amin-jhn]

Hey there! I am trying to convert my fine-tuned vibevoice into a gguf format for inference but cant find proper code or converter any where. The architecture is currently unsupported to any known converter like llama.cpp but I have already seen many variation of gguf-formated vibevoice. Can anyone help me with that?

[pwilkin]

So the status currently is that there's no support for VibeVoice in llama.cpp (but llama.cpp does support TTS architectures, it's a question of somebody adding support for that specific model).

What you have seen are probably repacks done by calcuis in his weirdly named "gguf-org project", but that project is not affiliated with ggml-org nor with llama.cpp in any way and from what I've analyzed the contents of those files, they are simply direct repacks of the original SafeTensors files, but with layers that are known to be quantizable by GGML converted to the respective quant format (with a side effect being that only a small percentage of the tensors that should be quantizable are really quantized, since the GGUF library does not know that for example model.acoustic_tokenizer.decoder.stages.0.0.ffn.linear1.weight should be a quantizable tensor). He then converts them back to F32 before running, so it's not a real GGML architecture, just a repack - not a very efficient one at that, since he unpacks all the quants to F32, so the memory usage will be the same as with the original model, just the quality will be lower (for example, the ComfyUI-GGUF solution, which supports GGUF quantization, does it by properly patching methods and dequantizing tensors on the fly as they're being used, so even though it does not utilize native quantized kernels for MAT_MUL like GGUF does, it actually allows to run those quants with a lower memory footprint).

TL;DR: You can try doing a repack in the same way, using quantize from the gguf-py library (a decent LLM coder will probably write that for you), but real VibeVoice support requires adding a proper MTMD model.

[amin-jhn]

Thank you so much for your detailed response!