|
| 1 | +# CacheBlend: : Fast Large Language Model Serving for RAG with Cached Knowledge Fusion |
| 2 | +<div align="center"> |
| 3 | + |
| 4 | + |
| 5 | + |
| 6 | +**🚀 Knowledge Cached Fusion Algorithm | 📄 EuroSys 2025 Paper ** |
| 7 | + |
| 8 | +[](https://github.com/ModelEngine-Group/unified-cache-management/blob/main/LICENSE) |
| 9 | +[](https://python.org) |
| 10 | + |
| 11 | +</div> |
| 12 | + |
| 13 | +## 🌟 What is CacheBlend? |
| 14 | + |
| 15 | +**CacheBlend** is a cached fusion system that combines multiple pre-computed KV caches, when their corresponding texts |
| 16 | +are concatenated in the LLM input. By selectively recomputing the KV cache values of a small fraction of tokens, |
| 17 | +CacheBlend reduces TTFT by 2.2 ~ 3.3× and increases throughput by 2.8 ~ 5× under negligible quality drop. |
| 18 | +### 🎯 Key Component |
| 19 | + |
| 20 | +- **🔍 Loading Controller**: the Loading Controller orchestrates which KV caches to load, where from, and how much recomputation is needed. |
| 21 | +- **⚡ KV Cache Store**: the KV Cache Store manages persistent storage, lookup, and eviction of precomputed KV caches keyed by text-chunk identity. |
| 22 | +- **🎛️ Cache Fusor**: the Fusor merges multiple chunk-level caches into one coherent, cross-attention–correct KV cache, using minimal recomputation. |
| 23 | + |
| 24 | +### 🔥 Key Results |
| 25 | +- **2.2 ~ 3.3× speedup** of TTFT and **2.8 ~ 5× increase** of throughput for long sequences |
| 26 | +- **Preserve High quality** no more than (1% ~ 3%) quality drop compared to full KV recompute |
| 27 | + |
| 28 | +## 🧠 Ucm Implementation |
| 29 | + |
| 30 | +### Native Block-Wise Chunk KV Cache Dump, Load, PostProcess and Recompute |
| 31 | +1. **🔐 Chunk Hash Encoding**: Similar as prefix hash encoder, hash all blocks in each chunk from the same hash meta beginning. |
| 32 | +2. **⚡ Combine Prefix Cache and Chunk Cache**: Since chunk cache and native prefix cache share the same hash space, ucm first performs prefix cache lookup to fetch fully reused cache and then conduct chunk cache lookup to fetch the candidate cache for blending. |
| 33 | +3. **🎯 Delta-Rope PostProcess**: Rectify loaded chunk cache according to their position in the new request. |
| 34 | +3. **🔍 Integrate Cache Blend and First Token Generation**: Construct compute mask and attention meta according to HKVD tokens, cache miss tokens and suffix tokens, then compute their kv cache in a single model forward stage. |
| 35 | +4. **🚀 Comprehensive Hook for LLM Forward Pipeline**: Based on ucm sparse module, blend module sparse the prefill tokens not only in attention stage but also in ffn, layer stage. |
| 36 | + |
| 37 | +## 🚀 Quick Start |
| 38 | + |
| 39 | +### Installation |
| 40 | + |
| 41 | +Blend is part of the UCM Sparse Attention module. For installation instructions, please refer to the [UCM's top-level README](https://github.com/ModelEngine-Group/unified-cache-management). Once UCM is installed, Blend is naturally supported by running the following example python scripts. |
| 42 | + |
| 43 | +```bash |
| 44 | +export ENABLE_SPARSE=TRUE |
| 45 | +export DATA_DIR=/home/data/kv_cache |
| 46 | +export MODEL_PATH=/home/models/mistralai/Mistral-7B-Instruct-v0.2 |
| 47 | +export BLEND_DATASET_PATH=/home/datasets/LongBench/data/2wikimqa.jsonl |
| 48 | +python <ucm-repo>/examples/offline_inference_blend.py |
| 49 | +``` |
| 50 | + |
| 51 | +### Basic Usage |
| 52 | +Similar to UCM's `offline_inference_esa.py` examples. We only need to specify `ucm_sparse_method` to be `Blend` and specify meta config, as shown below. |
| 53 | + |
| 54 | +```python |
| 55 | +... |
| 56 | +ktc = KVTransferConfig( |
| 57 | + kv_connector=name, |
| 58 | + kv_connector_module_path=module_path, |
| 59 | + kv_role="kv_both", |
| 60 | + kv_connector_extra_config={ |
| 61 | + "ucm_connectors": [ |
| 62 | + { |
| 63 | + "ucm_connector_name": "UcmNfsStore", |
| 64 | + "ucm_connector_config": { |
| 65 | + "storage_backends": data_dir, |
| 66 | + "kv_block_size": 33554432, |
| 67 | + }, |
| 68 | + } |
| 69 | + ], |
| 70 | + "load_only_first_rank": False, |
| 71 | + "ucm_sparse_config": { |
| 72 | + "Blend": { |
| 73 | + "chunk_end_token_id": chunk_end_token_id, |
| 74 | + "compute_meta": { |
| 75 | + "model.layers.1.self_attn.attn": { |
| 76 | + "ratio": 0.2, |
| 77 | + }, |
| 78 | + }, |
| 79 | + } |
| 80 | + }, |
| 81 | + "use_layerwise": True, |
| 82 | + }, |
| 83 | + ) |
| 84 | +... |
| 85 | +``` |
| 86 | + |
| 87 | +## 📊 Supported Models |
| 88 | +Llama-based models and Qwen-based models now are available |
| 89 | + |
| 90 | +## 🎓 Citation |
| 91 | + |
| 92 | +```bibtex |
| 93 | +@inproceedings{yao2025cacheblend, |
| 94 | + title={CacheBlend: Fast large language model serving for RAG with cached knowledge fusion}, |
| 95 | + author={Yao, Jiayi and Li, Hanchen and Liu, Yuhan and Ray, Siddhant and Cheng, Yihua and Zhang, Qizheng and Du, Kuntai and Lu, Shan and Jiang, Junchen}, |
| 96 | + booktitle={Proceedings of the Twentieth European Conference on Computer Systems}, |
| 97 | + pages={94--109}, |
| 98 | + year={2025} |
| 99 | +} |
| 100 | +``` |
| 101 | + |
| 102 | + |
| 103 | +--- |
| 104 | + |
| 105 | +<div align="center"> |
| 106 | + |
| 107 | +**🌟 Star [UCM](https://github.com/ModelEngine-Group/unified-cache-management) repository if you find KvComp useful!** |
| 108 | + |
| 109 | +</div> |
0 commit comments