Feature Request: Add Video Modality Support (Qwen2.5-VL) via llama-mtmd-cli

[deepshnv]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Add native video modality support to llama.cpp's multimodal pipeline (mtmd). Currently, llama.cpp supports vision primarily for static images (e.g., LLaVA, Qwen2-VL) via CLIP-based projectors. However, it lacks a built-in mechanism to process video inputs, a capability that is becoming standard in state-of-the-art Vision Language Models.

The recently released Qwen2.5-VL model is explicitly architected for video understanding. As noted in its technical report, it utilizes dynamic-resolution processing and absolute time encoding to handle videos of varying durations—from seconds to hours—enabling precise event localization. In Python ecosystems (HuggingFace Transformers, vLLM etc.), this is handled by accepting a video path and sampling parameters, then injecting specific tokens (e.g., <|vision_start|><|video_pad|><|vision_end|>) wrapping the temporal sequence.
Furthermore, this architecture serves as the foundation for Nvidia's Cosmos Reason1, a 7B-parameter model finetuned for physical AI, robotics, and chain-of-thought reasoning.

This feature request aims to replicate this video modality pipeline in llama.cpp, allowing users to pass video files directly to the inference engine, enabling tasks like video captioning and temporal reasoning on edge devices.

Motivation

With the release of high-performance open-weights models like Qwen2.5-VL, video understanding is moving from specialized proprietary APIs to local inference. Currently, Python-based frameworks (like vLLM and Transformers) support these video modalities out of the box.

Adding this support to llama.cpp is crucial for:

1. Feature Parity: Bridging the gap between C++ inference and Python research codebases (vLLM/Transformers).
2. Edge Application: Enabling local, privacy-preserving video analysis on consumer hardware (e.g., CPU, Apple Silicon, consumer GPUs) without heavy Python dependencies.
3. Specialized AI Support: Explicitly supporting Nvidia Cosmos Reason1, enabling developers to leverage its "Physical World" reasoning capabilities locally. This allows for embodied agent applications where the model reasons about physics and time from video input without relying on cloud APIs.

Possible Implementation

Add a working implementation locally that aligns with the Qwen2.5-VL technical report and mirrors the logic found in vLLM's implementation.

The implementation integrates into tools/mtmd and clip.cpp with the following key components:

1. Frame Extraction (External FFMPEG)

As per the Coding Guidelines, we can avoid adding large third-party dependencies to the codebase by linking against libavcodec/libavformat directly. Instead the implementation should rely on the user having the ffmpeg CLI tool installed and available in their system PATH.

• Logic: The system checks for ffmpeg, creates a temporary directory, extracts frames at a specified FPS (defaulting to 1.0 or user-defined) via a command-line call, and loads the resulting frames as a batch.
• Benefit: Keeps the build system simple and dependency-free while leveraging the industry standard for media processing.

2. Visual Patch Extraction & Temporal Merging

The implementation handles the Qwen2.5-VL specific "super-frame" logic:

• Temporal Patching: It merges consecutive frames (Temporal Patch Size = 2) into a single "super-frame" with 6 channels (2 frames × 3 RGB channels).
• Conv3D Decomposition: In clip.cpp, the Conv3D operation (2×14×14) is mathematically decomposed into two separate Conv2D operations on the temporal slices, which are then summed. This avoids implementing a full 3D convolution operator (which is currently not support in the ggml backend) while maintaining mathematical equivalence.

3. 3D M-RoPE & Absolute Time Encoding

Qwen2.5-VL uses a 3D Rotary Positional Embedding to encode time alongside spatial dimensions:

• Structure: [text_pos, temporal_idx, height_pos, width_pos]
• Temporal Index: Calculated as round(frame_idx * seconds_per_grid * tokens_per_second).
• Fallback: The system detects if the input is a video frame or static image and gracefully degrades to standard 2D M-RoPE for regular images.

4. Nvidia Cosmos Reason1 Specifics

To support Cosmos Reason1, I have added specific handling in arg.cpp and mtmd-cli.cpp:

• Custom Chat Template: Added a cosmos template that handles the model's specific Chain-of-Thought requirements. It formats prompts with specific system instructions (<|im_start|>system...) and supports a "reasoning" mode that parses <think> blocks.
• Marker Mapping: The implementation correctly maps the distinct vision markers used by the Cosmos finetunes:
  • Video: <|vision_start|><|video_pad|><|vision_end|>
  • Image: <|vision_start|><|image_pad|><|vision_end|>

5. Pipeline Integration

• Vision Transformer: In clip.cpp, implement new methods i.e. build_qwen25_vl() and clip_image_batch_encode_video() to handle the "super-frame" batches, applying RMSNorm and Window Attention where appropriate.
• Vision-Language Merger: The spatial merging (2×2 pooling) and projection adapter layers are applied to the output tokens before concatenation with the text prompt.
• CLI Arguments: New arguments added to common for ease of use: --video <path>, --video-fps <N>, -vmaxf <N> (max frames), --ts-per-grid (seconds per temporal grid for 3D M-RoPE), --chat-template cosmos etc.

[bssrdf]

• Conv3D Decomposition: In clip.cpp, the Conv3D operation (2×14×14) is mathematically decomposed into two separate Conv2D operations on the temporal slices, which are then summed. This avoids implementing a full 3D convolution operator (which is currently not support in the ggml backend) while maintaining mathematical equivalence.

Conv3D is supported now by either IM2COL_3D+GEMM or implicit GEMM (PR to be merged)

[ngxson]

Most of the content in the description is generated by AI, as such, many points are incorrect.

To clarify:

• We are working on video input implementation
• Conv3D is not needed, it is already implemented as two 2D operations and was remain the same since Qwen2-VL

Please note that, you should explicitly state if you use AI to create issues / PRs to avoid any confusions.

[deepshnv]

@ngxson, Explicit use of AI while creating this issue: I used cursor to create a formatted description from the rough notes I had when doing a local implementation of the video input support, though I did review it before posting.

Sorry I didn't know others were working on video modality. I had to implement this change for a project and thought it would be useful to contribute back.
The reason for starting this issue was to share my implementation and getting it reviewed for a PR.

[deepshnv]

@ggerganov @ngxson, My changes to implement video modality support(specifically for qwen2/2.5-vl) can be found here: https://github.com/deepshnv/llama.cpp/tree/qwen2vl_video_modality
Use of AI: I took some help from the cursor code-editor, mainly to get some chunk of my solution implemented at the right places in the llama.cpp code-base and for comment purposes. I have done due diligence and manual review while accepting any AI generated code and also tested the the runtime once the implementation was completed.

Please let me know if this looks okay to open a PR, also please feel free to use some/all parts of this implementation as you see fit!

Thanks

[ngxson]

Generally we don't recommend pushing a PR mostly written by AI. It will only slow down the works of maintainers.

Please patiently wait. The feature is already planned - there is no need to be rush.

[deepshnv]

Please let me clarify in case there is a misunderstanding. I used AI to draft and format this issue. But the code itself is mostly my work with some help from AI for structure. Wherever I used AI prescribed code, I have made sure to vet and fully understand the logic before adding to llama.cpp (suitably renaming variables and adding comments as needed to be consistent with llama.cpp programming practice). I have also performed detailed comparisons vs vllm for video inference(across multiple videos), and found things working correctly.
I would appreciate if you can spare some time to skim through the implementation once and see if there is anything concerning? I will leave it to you and the other maintainers to decide if and how much of my code to use.

Thanks

[ngxson]

I suggest you to refer to this PR for more context.

I already skimmed though your PR but the main problem is that it is design only to work with Qwen. Something like MiniCPM-V won't fit well here.

As I said, let's be patient and include the feature one thing at a time. I prefer take time to refine the API to allow even more video models to be compatible in the future, not just Qwen.

[ngxson]

Also, one more thing to note is that we're also planning to allow processing both image+audio input at the same time (Qwen-Omni), which allows true video understanding.

The assumption is that this may become a trend in the future (more models will follow the trend) as a video is technically incomplete without both vision+audio understanding. If the API is well-designed, support these future models can become much easier

I'll update more details for more visibility in another issue.

[deepshnv]

Also, one more thing to note is that we're also planning to allow processing both image+audio input at the same time (Qwen-Omni), which allows true video understanding.

The assumption is that this may become a trend in the future (more models will follow the trend) as a video is technically incomplete without both vision+audio understanding. If the API is well-designed, support these future models can become much easier

I'll update more details for more visibility in another issue.

Ohh I see, that's a fair point. Thanks for the detailed response

[FMayran]

Just taking advantage of this already opened PR to add my personal whichlist for a video API. (especially targeted at @ngxson )

For me, when talking about video, there is a very important factor I don't see discussed anywhere. I don't think we should provide llamacpp with the whole video (all frames) in one go. Rather, we should provide a frame pump callback, a function which asks for the next frame once the current one has been encoded (with a special convention such as returning nullptr = no more frames). This has two key advantages: first, the RAM consumption is much lower, as we can decode the next frame on the fly. Second, and combined with KVcache checkpointing (via save/load to an in-memory buffer), we could process live video from a camera. This would be a very cool feature, and a differenciating factor from competing APIs such as vLLM

[ngxson]

there is a very important factor I don't see discussed anywhere.

I would like to spend time to write a detailed issue for visibility, but still haven't yet found a time slot. Will have a look a bit later this week.

I don't think we should provide llamacpp with the whole video

Yes, I already acknowledge this problem. But currently, the preprocessing pipeline need to store F32 (float type) image in RAM for all images, so this will make the idea of callback (aka streaming) not as efficient.

Please note that it was designed this way for historical reason (before I even joined llama.cpp). Also, I believe that pytorch (transformers library) also work this way.

To enable true streaming processing, the preprocessing pipeline need to be redesigned as a whole. But I think this can be done in a later stage. Still, the important things is that the API need to be ready in advance for such future refactoring

[FMayran]

@ngxson I understand the issues. I think the framepump callback still stands its ground on the API side: what matters is that within the LLM/VLM encoding, it sees one frame at a time. Whether this frame in practice just links memory from what is CURRENTLY a single large buffer is not an issue (the LLM/VLM should not know or assume it comes from a large buffer). So, later we can replace the decoding backend for something else which decodes one frame at a time (or bypass it entirely, in my case I have a huge application which uses llamacpp as a library, and has opencv available as a video decoder, with cv::cap. Currently, I feed images to llamacpp via my own applciation, and would like to feed videos the same way)

Edit:
my idea of a stable API:

void mtmd_add_video(std::function<mtmd_bitmap(int frame_number, [perhaps the context,?])> get_frame);

and then, supposing your large float32 buffer is float32* buf;, you just call:

mtmd_add_video([&](int frame_number){
  mtmd_bitmap bm;
  if (frame_number >= video_size)
    return bm;//finished
  bm.nx = w;
  bm.ny = h;
  bm.data.resize(w*h*4*3);
  std::copy(buf+ w*h * 3*frame_number, buf + w * h * 3*(frame_number + 1), reinterpret_cast<float*>(bm.data.data()));
  return bm;
}

Edit2 : forgot RGB