RFC: ensure byte order consistency in GGUF and llama.cpp across LE and BE hosts

[zv-io]

This relates to #3552 and #3957.

I want to start a broader discussion about whether and how to support endianness (byte order) differences between some host platforms and models, and to push for a coherent and consistent path forward, to accomplish these objectives:

1. Avoid conversion/bandwidth/storage waste;
2. Ensure that models run correctly on LE and BE hosts;
3. Ensure that any effort put into developing format specifications, conversion tools, and llama.cpp itself is well spent, prioritizing accuracy and correctness and avoiding future code churn or incompatibilities;
4. Understand any performance implications.

While the codebase as of at least 2376b77 will compile on big-endian targets (e.g. ppc64), it will quickly encounter errors while loading little-endian .gguf models, as early as the built-in test suite:

$ llama-cli --version
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (CPU)
version: 11624 (907c5c34d)
built with gcc (REDACTED 13.3.0) 13.3.0 for powerpc64-REDACTED-linux-musl
...
20/39 Test #20: test-tokenizer-1-llama-spm ........***Failed    0.01 sec
main : reading vocab from: '/usr/src/packages/user/llama-cpp/src/llama.cpp-b7058/models/ggml-vocab-llama-spm.gguf'
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (CPU)
gguf_init_from_file_impl: failed to load model: this GGUF file version 50331648 is extremely large, is there a mismatch between the host and model endianness?
gguf_init_from_file_impl: failed to read header
llama_model_load: error loading model: llama_model_loader: failed to load model from /usr/src/packages/user/llama-cpp/src/llama.cpp-b7058/models/ggml-vocab-llama-spm.gguf
llama_model_load_from_file_impl: failed to load model
main: error: failed to load vocab '/usr/src/packages/user/llama-cpp/src/llama.cpp-b7058/models/ggml-vocab-llama-spm.gguf'

This assertion (garbage file version) fails due to a check which assumes file version byte order:

if (ok && (ctx->version & 0x0000FFFF) == 0x00000000) {
    GGML_LOG_ERROR("%s: failed to load model: this GGUF file version %" PRIu32 " is extremely large, is there a mismatch between the host and model endianness?\n", __func__, ctx->version);
    ok = false;
}

The on-disk start of the file is:

# xxd original.gguf | head -n 1
00000000: 4747 5546 0300 0000 3200 0000 0000 0000  GGUF....2....... <-- real LE model
                    ^^^^^^^^^
                    (version)

If the proposal to use GGUF magic for LE files and FUGG magic for BE files is to be implemented:

• Transmitting and storing one copy of each would require twice the disk space while providing no new information.

• Conversion tools will require maintenance as the format evolves, and will require robust validation to ensure they work flawlessly.

• Robust file format specifications will need to be laid out, including specifying whether all, or only a subset of, fields/values/parameters will need to be interpreted as LE or BE. I cannot speak for performance about model loading times or whether conversion should happen on the fly.

If we patch the GGUF files in place such that:

# xxd patched.gguf | head -n 1
00000000: 4747 5546 0000 0003 0000 0000 0000 0032  GGUF...........2 <-- hypothetical BE model

then it will be difficult to ensure such a conversion is done correctly, as the conversion tool(s) themselves will also need to be endian-correct whether they're used on an LE or BE host. This will not prevent cross-endian GGUF files from failing to run on the opposite host, and likely many tools will choke as well.

I believe it is a software (inference engine) issue, not strictly a file format issue, to ensure that data are encoded and interpreted correctly. The file format (data exchange vehicle) should probably be consistent (effectively serialized) regardless of the byte order of the system that produced the GGUF. In other words, host byte order should not affect the data written to disk. The file format should specify how the data are to be interpreted.

Similarly, llama.cpp should behave the same (modulo any performance penalties) regardless of the input file. Suppose we fix the loader so that it is endian-agnostic, now other such assumptions are revealed:

...
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)                              
Assertion failed: scale > 0.0f (/usr/src/packages/user/llama-cpp/src/llama.cpp-b7058/ggml/src/ggml-cpu/ops.cpp: ggml_compute_forward_rms_norm_f32: 3689)                                                                                                                        
Aborted

While BE systems are not nearly as common or popular as LE systems, they do exist. My proposal is to fix all byte order assumptions in the llama.cpp codebase but I want to know if this topic has been explored in depth and whether/where to put my and others' effort.

[awilfox]

Note that whisper.cpp had ggml-org/whisper.cpp#398 and ggml-org/whisper.cpp#2816 successfully merged to have big-endian support with a similar theory of implementation.

[taronaeo]

While the codebase as of at least 2376b77 will compile on big-endian targets (e.g. ppc64), it will quickly encounter errors while loading little-endian .gguf models, as early as the built-in test suite:

1. As of devops: add s390x & ppc64le CI #15925, test suites for Big-Endian platforms use a pre-converted Big-Endian model as shown here:

https://github.com/taronaeo/llama.cpp-s390x/blob/8ab049129c092b13c728e809ce125ef94bae57bd/.github/workflows/build.yml#L262-L269

If you intend on running the same test on your system, you should point the test to the Big-Endian test model.

If we patch the GGUF files in place such that:

xxd patched.gguf | head -n 1

00000000: 4747 5546 0000 0003 0000 0000 0000 0032 GGUF...........2 <-- hypothetical BE model
then it will be difficult to ensure such a conversion is done correctly, as the conversion tool(s) themselves will also need to be endian-correct whether they're used on an LE or BE host. This will not prevent cross-endian GGUF files from failing to run on the opposite host, and likely many tools will choke as well.

2. Discerning Little-Endian and Big-Endian model byte-order is currently done by reading the byte-order of the version, which you have pointed out.

We have an existing tool (gguf_convert_endian.py) that uses numpy to byteswap the models and it seem to have performed fine for the supported datatypes/structures.

3. There was actually a proposal by @AlekseiNikiforovIBM to load Little-Endian models on Big-Endian systems on-the-fly but wasn't accepted: Allow s390x to load little endian models unmodified #11234

[zv-io]

1. Thanks for the pointer. I can confirm that, unmodified, llama.cpp can run this model on a BE host (ppc64 in this case).

2. This behavior will only detect if the file matches the host platform but does not ensure validity of the rest of the file, which I'm not clear on whether is intended to be fully byte-swapped, or only certain sections, or if there's an explicit mix of "LE + Host" endian data. The closest I can find to a file specification is

3. Ah, thank you. I had not seen this. For reference, some of those changes seem to have been integrated in this project. Based on the discussion in Allow s390x to load little endian models unmodified #11234 it seems this project's position is to pretend BE systems don't exist and to punt model conversion to users of those systems without concern to the broader ecosystem, the least painful route will be for model providers to convert models on demand, assuming there is a robust conversion toolkit available.

4. The closest I can find to a GGML or GGUF specification is https://github.com/ggml-org/ggml/blob/master/docs/gguf.md; do I understand this correctly, or is there a canonical specification elsewhere? If the endianness of all of the fields/data within such files can be fully specified, it makes writing conversion tools much easier.

I will do some testing on both LE and BE systems with existing conversion tools and provide some additional thoughts/commentary. I assume that, realistically, all future development of this project and other downstream GGUF consumers will only test on LE systems without dedicated BE-interested maintainers. It's clear that access to BE hardware is not the issue.

[CISC]

I assume that, realistically, all future development of this project and other downstream GGUF consumers will only test on LE systems without dedicated BE-interested maintainers. It's clear that access to BE hardware is not the issue.

Thanks to @taronaeo et al. at IBM we have recently added big endian CIs running the full test suite, which means we are now in a much better position to maintain this.

[taronaeo]

2. This behavior will only detect if the file matches the host platform but does not ensure validity of the rest of the file, which I'm not clear on whether is intended to be fully byte-swapped, or only certain sections, or if there's an explicit mix of "LE + Host" endian data. The closest I can find to a file specification is

For platforms which are Big-Endian, we would usually denote the GGUF file with the -be.Q*.gguf suffix, f.ex.: https://huggingface.co/taronaeo/Granite-3.3-2B-Instruct-BE-GGUF/tree/main. There doesn't seem to be a standard on how to document endianness though, so this should be worked on. (CC: @CISC?)

I may be wrong, if I recall correctly, all the tensor data (except strings) within the GGUF file is byteswapped from LE to BE endianness, so there isn't a specific "section".

3. Ah, thank you. I had not seen this. For reference, some of those changes seem to have Enable automatic endianness conversion, SIMD & MASS support zopencommunity/llamacppport#4. Based on the discussion in Allow s390x to load little endian models unmodified #11234 it seems this project's position is to pretend BE systems don't exist and to punt model conversion to users of those systems without concern to the broader ecosystem, the least painful route will be for model providers to convert models on demand, assuming there is a robust conversion toolkit available.

That project seem to be maintained by our IBMers, with support mainly for z/OS which is very different from zLinux. Here we explicitly support zLinux only :) (which is technically the same as Linux)

4. The closest I can find to a GGML or GGUF specification is https://github.com/ggml-org/ggml/blob/master/docs/gguf.md; do I understand this correctly, or is there a canonical specification elsewhere? If the endianness of all of the fields/data within such files can be fully specified, it makes writing conversion tools much easier.

That is the correct specification. There is currently no field for containing endianness data, but what you can do is to actually include a metadata f.ex., general.endianness: LITTLE to help inform the model's endianness.

[CISC]

For platforms which are Big-Endian, we would usually denote the GGUF file with the -be.Q*.gguf suffix, f.ex.: https://huggingface.co/taronaeo/Granite-3.3-2B-Instruct-BE-GGUF/tree/main. There doesn't seem to be a standard on how to document endianness though, so this should be worked on. (CC: @CISC?)

Unfortunately not, it's only inferred from the version field.

[zv-io]

@philpax as the author of ggml-org/ggml#302 would you be in a position to comment? I am doing some conversion and performance tests and trying to understand various stakeholders/consumers in the broader ecosystem.

[philpax]

Hi there! I'm afraid I don't have much of an opinion on this; I haven't done much work on the format after initially drafting the spec.

As you've mentioned, I called this out as an issue in #3957, but didn't get much movement there; I don't personally interact with any big-endian systems, so it wasn't a matter of pressing importance to me. I do think that this is something that should be standardised, though!

[zv-io]

Thanks; in this case @awilfox do you want to collaborate here to clean up the spec, implementations, documentation, and testing? We have the hardware.

[philpax]

Something worth mentioning is that the initial proposal to have big-endian-specific files was born out of the desire to keep the files mmap-able, especially for CPU inference: without this, inference engines will have to pay the cost of conversion on startup and/or keep the model entirely resident within memory.

That being said, I think not being able to load LE models at all on BE platforms is counterproductive, especially as the vast majority of models target LE. I'd suggest adding support for loading and running LE, and offering BE models as a one-time conversion for users that require reduced latency/the ability to stream the model from disk.

[zv-io]

@philpax the tradeoff for mmap is storing two copies of the model, or having an on-the-fly conversion tool at the model repositories' edge nodes (since I doubt they'd want to waste disk space on a niche user base).

Given the extremely limited big-endian user base, if the file-on-disk format is standardized to little-endian and the spec is better defined, then perhaps it is acceptable for big-endian users to either convert LE models to host-native byte order before usage using a canonical conversion tool (which could be the same as what model hosts would use), or to do so on demand during loading at the cost of losing the ability to use mmap.

Anything will be an improvement over the current state of affairs, both from a user experience point of view and from a security point of view.

[philpax]

Aye, that's what I was describing: one-time conversion at the point of use to convert the canonical LE GGUF to BE - for those users where the tradeoffs matter - in addition to being able to load LE normally without mmap. Agreeing on any solution would be an improvement, though, yes.