mtmd: Add DeepSeekOCR Support

[sfallah]

Feature Request: #16676

Make sure to read the contributing guidelines before submitting a PR

GGUF Models

sabafallah/DeepSeek-OCR-GGUF

deepseek-ocr-f32.gguf

mmproj-deepseek-ocr-f32.gguf

Running the Model

Build llama.cpp (Mac)

cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --config Release

Running llama-mtmd-cli

build/bin/llama-mtmd-cli \
-m gguf_models/deepseek-ai/deepseek-ocr-f32.gguf \
--mmproj gguf_models/deepseek-ai/mmproj-deepseek-ocr-f32.gguf \
--image tmp/mtmd_test_data/Deepseek-OCR-2510.18234v1_page1.png \
-p "<|grounding|>Convert the document to markdown." \
--chat-template deepseek \

[ngxson]

IMO these modes can look quite confusing for end-users.

I already seen your logic where you calculate the area to automatically determine the best resolution, it looks good enough.

So, I think we can better remove the argument and make everything automatic.

[bluebread]

ok. I'll remove it later.

[bluebread]

@ngxson I've encountered an issue with batching images. In order to handle images much larger than 1280x1280, DeepSeek-OCR crops them into 640x640 (gundam) or 1024x1024 (gundam master) subimages as local views. However, the current framework doesn't support batching multiple images. Technically, it shouldn't be too difficult to add batch support, but I'm concerned about introducing new bugs and affecting other models. Do you have any suggestions?

[ngxson]

IIRC it should be the same logic as llava uhd or minicpm-v where image is cropped into smaller sub-images

batching is not yet supported, but does all sub-images need to be on the same batch?

otherwise, what we can do is to extend the clip_image_f32 to include a notion of "nz":

struct clip_image_f32 {
    int nx;
    int ny;
    int nz; // can be > 1 for deepseek ocr

    std::vector<float> buf;
};

And memory layout corresponding to what you need on cgraph (to avoid another ggml_permute for example)

[bluebread]

@sfallah Could you please mark this PR as ready for review and update the llama-mtmd-cli command for testing DeepSeek-OCR (because I removed --dsocr-mode argument)? Also, I have run the CI locally, and it failed on "27 - test-thread-safety". I guess this failure should be unrelated to the changes made in this PR. Here is the log: ci.txt