Skip to content

Commit 2bb94b9

Browse files
author
Paweł Kędzia
committed
Update README – reformat description, rename **Plugins** to **llm_router_plugins**, tidy the environment‑variable table alignment, and label the dynamic_weighted strategy as beta.
1 parent f513cee commit 2bb94b9

File tree

1 file changed

+33
-33
lines changed

1 file changed

+33
-33
lines changed

‎README.md‎

Lines changed: 33 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
## llm‑router
22

3-
**LLM Router** is a service that can be deployed on‑premises or in the cloud.
4-
It adds a layer between any application and the LLM provider. In real time it controls traffic,
5-
distributes a load among providers of a specific LLM, and enables analysis of outgoing requests
6-
from a security perspective (masking, anonymization, prohibited content).
7-
It is an open‑source solution (Apache 2.0) that can be launched instantly by running
3+
**LLM Router** is a service that can be deployed on‑premises or in the cloud.
4+
It adds a layer between any application and the LLM provider. In real time it controls traffic,
5+
distributes a load among providers of a specific LLM, and enables analysis of outgoing requests
6+
from a security perspective (masking, anonymization, prohibited content).
7+
It is an open‑source solution (Apache 2.0) that can be launched instantly by running
88
a ready‑made image in your own infrastructure.
99

1010
- **llm_router_api** provides a unified REST proxy that can route requests to any supported LLM backend (
@@ -14,9 +14,9 @@ a ready‑made image in your own infrastructure.
1414
handling and a rich exception hierarchy, letting developers focus on application logic rather than raw HTTP calls.
1515
- **llm_router_web** offers ready‑to‑use Flask UIs – an anonymizer UI that masks sensitive data and a configuration
1616
manager for model/user settings – demonstrating how to consume the router from a browser.
17-
- **Plugins** (e.g., the **fast_masker** plugin) deliver a rule‑based text anonymisation engine with a comprehensive set
18-
of Polish‑specific masking rules (emails, IPs, URLs, phone numbers, PESEL, NIP, KRS, REGON, monetary amounts, dates,
19-
etc.) and an extensible architecture for custom rules and validators.
17+
- **llm_router_plugins** (e.g., the **fast_masker** plugin) deliver a rule‑based text anonymisation engine with
18+
a comprehensive set of Polish‑specific masking rules (emails, IPs, URLs, phone numbers, PESEL, NIP, KRS, REGON,
19+
monetary amounts, dates, etc.) and an extensible architecture for custom rules and validators.
2020

2121
All components run on Python 3.10+ using `virtualenv` and require only the listed dependencies, making the suite easy to
2222
install, extend, and deploy in both development and production environments.
@@ -149,30 +149,30 @@ docker run \
149149

150150
### 3️⃣ Optional configuration (via environment)
151151

152-
| Variable | Description | Default |
153-
|---------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------|
154-
| `LLM_ROUTER_PROMPTS_DIR` | Directory containing predefined system prompts. | `resources/prompts` |
155-
| `LLM_ROUTER_MODELS_CONFIG` | Path to the models configuration JSON file. | `resources/configs/models-config.json` |
156-
| `LLM_ROUTER_DEFAULT_EP_LANGUAGE` | Default language for endpoint prompts. | `pl` |
157-
| `LLM_ROUTER_TIMEOUT` | Timeout (seconds) for llm-router API calls. | `0` |
158-
| `LLM_ROUTER_EXTERNAL_TIMEOUT` | Timeout (seconds) for external model API calls. | `300` |
159-
| `LLM_ROUTER_LOG_FILENAME` | Name of the log file. | `llm-router.log` |
160-
| `LLM_ROUTER_LOG_LEVEL` | Logging level (e.g., INFO, DEBUG). | `INFO` |
161-
| `LLM_ROUTER_EP_PREFIX` | Prefix for all API endpoints. | `/api` |
162-
| `LLM_ROUTER_MINIMUM` | Run service in proxy‑only mode (boolean). | `False` |
163-
| `LLM_ROUTER_IN_DEBUG` | Run server in debug mode (boolean). | `False` |
164-
| `LLM_ROUTER_BALANCE_STRATEGY` | Strategy used to balance routing between LLM providers. Allowed values are `balanced`, `weighted`, `dynamic_weighted` and `first_available` as defined in `constants_base.py`. | `balanced` |
165-
| `LLM_ROUTER_REDIS_HOST` | Redis host for load‑balancing when a multi‑provider model is available. | `<empty string>` |
166-
| `LLM_ROUTER_REDIS_PORT` | Redis port for load‑balancing when a multi‑provider model is available. | `6379` |
167-
| `LLM_ROUTER_SERVER_TYPE` | Server implementation to use (`flask`, `gunicorn`, `waitress`). | `flask` |
168-
| `LLM_ROUTER_SERVER_PORT` | Port on which the server listens. | `8080` |
169-
| `LLM_ROUTER_SERVER_HOST` | Host address for the server. | `0.0.0.0` |
170-
| `LLM_ROUTER_SERVER_WORKERS_COUNT` | Number of workers (used in case when the selected server type supports multiworkers) | `2` |
171-
| `LLM_ROUTER_SERVER_THREADS_COUNT` | Number of workers threads (used in case when the selected server type supports multithreading) | `8` |
172-
| `LLM_ROUTER_SERVER_WORKER_CLASS` | If server accepts workers type, its able to set worker class by this environment. | `None` |
173-
| `LLM_ROUTER_USE_PROMETHEUS` | Enable Prometheus metrics collection.** When set to `True`, the router registers a `/metrics` endpoint exposing Prometheus‑compatible metrics for monitoring. | `False` |
174-
| `LLM_ROUTER_FORCE_ANONYMISATION` | Enable whole payload anonymisation. Each key and value is aut-anonymized before sending to model provider. | `False` |
175-
| `LLM_ROUTER_ENABLE_GENAI_ANONYMIZE_TEXT_EP` | Enable builtin endpoint `/api/anonymize_text_genai` which uses genai to anonymize text | `False` |
152+
| Variable | Description | Default |
153+
|---------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------|
154+
| `LLM_ROUTER_PROMPTS_DIR` | Directory containing predefined system prompts. | `resources/prompts` |
155+
| `LLM_ROUTER_MODELS_CONFIG` | Path to the models configuration JSON file. | `resources/configs/models-config.json` |
156+
| `LLM_ROUTER_DEFAULT_EP_LANGUAGE` | Default language for endpoint prompts. | `pl` |
157+
| `LLM_ROUTER_TIMEOUT` | Timeout (seconds) for llm-router API calls. | `0` |
158+
| `LLM_ROUTER_EXTERNAL_TIMEOUT` | Timeout (seconds) for external model API calls. | `300` |
159+
| `LLM_ROUTER_LOG_FILENAME` | Name of the log file. | `llm-router.log` |
160+
| `LLM_ROUTER_LOG_LEVEL` | Logging level (e.g., INFO, DEBUG). | `INFO` |
161+
| `LLM_ROUTER_EP_PREFIX` | Prefix for all API endpoints. | `/api` |
162+
| `LLM_ROUTER_MINIMUM` | Run service in proxy‑only mode (boolean). | `False` |
163+
| `LLM_ROUTER_IN_DEBUG` | Run server in debug mode (boolean). | `False` |
164+
| `LLM_ROUTER_BALANCE_STRATEGY` | Strategy used to balance routing between LLM providers. Allowed values are `balanced`, `weighted`, `dynamic_weighted` (beta), `first_available` and `first_available_optim` as defined in `constants_base.py`. | `balanced` |
165+
| `LLM_ROUTER_REDIS_HOST` | Redis host for load‑balancing when a multi‑provider model is available. | `<empty string>` |
166+
| `LLM_ROUTER_REDIS_PORT` | Redis port for load‑balancing when a multi‑provider model is available. | `6379` |
167+
| `LLM_ROUTER_SERVER_TYPE` | Server implementation to use (`flask`, `gunicorn`, `waitress`). | `flask` |
168+
| `LLM_ROUTER_SERVER_PORT` | Port on which the server listens. | `8080` |
169+
| `LLM_ROUTER_SERVER_HOST` | Host address for the server. | `0.0.0.0` |
170+
| `LLM_ROUTER_SERVER_WORKERS_COUNT` | Number of workers (used in case when the selected server type supports multiworkers) | `2` |
171+
| `LLM_ROUTER_SERVER_THREADS_COUNT` | Number of workers threads (used in case when the selected server type supports multithreading) | `8` |
172+
| `LLM_ROUTER_SERVER_WORKER_CLASS` | If server accepts workers type, its able to set worker class by this environment. | `None` |
173+
| `LLM_ROUTER_USE_PROMETHEUS` | Enable Prometheus metrics collection.** When set to `True`, the router registers a `/metrics` endpoint exposing Prometheus‑compatible metrics for monitoring. | `False` |
174+
| `LLM_ROUTER_FORCE_ANONYMISATION` | Enable whole payload anonymisation. Each key and value is aut-anonymized before sending to model provider. | `False` |
175+
| `LLM_ROUTER_ENABLE_GENAI_ANONYMIZE_TEXT_EP` | Enable builtin endpoint `/api/anonymize_text_genai` which uses genai to anonymize text | `False` |
176176

177177
### 4️⃣ Run the REST API
178178

@@ -208,7 +208,7 @@ and reliable routing of requests. The available strategies are:
208208
characteristics, and you want to prioritize certain providers without needing dynamic adjustments.
209209
* **Implementation:** Implemented in `llm_router_api.base.lb.weighted.WeightedStrategy`.
210210

211-
### 3. `dynamic_weighted`
211+
### 3. `dynamic_weighted` (beta)
212212

213213
* **Description:** An extension of the `weighted` strategy. It not only uses weights
214214
but also tracks the latency between successive selections of the same provider.

0 commit comments

Comments
 (0)