You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using [vLLM](https://docs.vllm.ai/en/latest/). **All scripts in this repository runs natively on the Vector Institute cluster environment**. To adapt to other environments, update [`launch_server.sh`](vec_inf/launch_server.sh), [`vllm.slurm`](vec_inf/vllm.slurm), [`multinode_vllm.slurm`](vec_inf/multinode_vllm.slurm) and [`models.csv`](vec_inf/config/models.yaml) accordingly.
11
+
This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using [vLLM](https://docs.vllm.ai/en/latest/). **All scripts in this repository runs natively on the Vector Institute cluster environment**. To adapt to other environments, update the environment variables in [`cli/_helper.py`](vec_inf/cli/_helper.py), [`cli/_config.py`](vec_inf/cli/_config.py), [`vllm.slurm`](vec_inf/vllm.slurm), [`multinode_vllm.slurm`](vec_inf/multinode_vllm.slurm) and [`models.yaml`](vec_inf/config/models.yaml) accordingly.
12
12
13
13
## Installation
14
14
If you are using the Vector cluster environment, and you don't need any customization to the inference server environment, run the following to install package:
@@ -22,8 +22,7 @@ Otherwise, we recommend using the provided [`Dockerfile`](Dockerfile) to set up
22
22
23
23
### `launch` command
24
24
25
-
The `launch` command allows users to deploy a model as a slurm job. If the job successfully launches, a URL endpoint is exposed for
26
-
the user to send requests for inference.
25
+
The `launch` command allows users to deploy a model as a slurm job. If the job successfully launches, a URL endpoint is exposed for the user to send requests for inference.
27
26
28
27
We will use the Llama 3.1 model as example, to launch an OpenAI compatible inference server for Meta-Llama-3.1-8B-Instruct, run:
29
28
@@ -36,7 +35,7 @@ You should see an output like the following:
36
35
37
36
#### Overrides
38
37
39
-
Models that are already supported by `vec-inf` would be launched using the [default parameters](vec_inf/config/models.yaml). You can override these values by providing additional parameters. Use `vec-inf launch --help` to see the full list of parameters that can be
38
+
Models that are already supported by `vec-inf` would be launched using the cached configuration or [default configuration](vec_inf/config/models.yaml). You can override these values by providing additional parameters. Use `vec-inf launch --help` to see the full list of parameters that can be
40
39
overriden. For example, if `qos` is to be overriden:
You can also launch your own custom model as long as the model architecture is [supported by vLLM](https://docs.vllm.ai/en/stable/models/supported_models.html), and make sure to follow the instructions below:
49
-
* Your model weights directory naming convention should follow `$MODEL_FAMILY-$MODEL_VARIANT`.
48
+
* Your model weights directory naming convention should follow `$MODEL_FAMILY-$MODEL_VARIANT` ($MODEL_VARIANT is OPTIONAL).
50
49
* Your model weights directory should contain HuggingFace format weights.
51
-
* You should create a custom configuration file for your model and specify its path via setting the environment variable `VEC_INF_CONFIG`
52
-
Check the [default parameters](vec_inf/config/models.yaml) file for the format of the config file. All the parameters for the model
53
-
should be specified in that config file.
50
+
* You should specify your model configuration by:
51
+
* Creating a custom configuration file for your model and specify its path via setting the environment variable `VEC_INF_CONFIG`. Check the [default parameters](vec_inf/config/models.yaml) file for the format of the config file. All the parameters for the model should be specified in that config file.
52
+
* Using launch command options to specify your model setup.
54
53
* For other model launch parameters you can reference the default values for similar models using the [`list` command ](#list-command).
55
54
56
55
Here is an example to deploy a custom [Qwen2.5-7B-Instruct-1M](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-1M) model which is not
NOTE: The above screenshot does not represent the full list of models supported.
136
133
137
134
You can also view the default setup for a specific supported model by providing the model name, for example `Meta-Llama-3.1-70B-Instruct`:
138
135
```bash
@@ -143,9 +140,39 @@ vec-inf list Meta-Llama-3.1-70B-Instruct
143
140
`launch`, `list`, and `status` command supports `--json-mode`, where the command output would be structured as a JSON string.
144
141
145
142
## Send inference requests
146
-
Once the inference server is ready, you can start sending in inference requests. We provide example scripts for sending inference requests in [`examples`](examples) folder. Make sure to update the model server URL and the model weights location in the scripts. For example, you can run `python examples/inference/llm/completions.py`, and you should expect to see an output like the following:
147
-
> {"id":"cmpl-c08d8946224747af9cce9f4d9f36ceb3","object":"text_completion","created":1725394970,"model":"Meta-Llama-3.1-8B-Instruct","choices":[{"index":0,"text":" is a question that many people may wonder. The answer is, of course, Ottawa. But if","logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":8,"total_tokens":28,"completion_tokens":20}}
148
-
143
+
Once the inference server is ready, you can start sending in inference requests. We provide example scripts for sending inference requests in [`examples`](examples) folder. Make sure to update the model server URL and the model weights location in the scripts. For example, you can run `python examples/inference/llm/chat_completions.py`, and you should expect to see an output like the following:
144
+
145
+
```json
146
+
{
147
+
"id":"chatcmpl-387c2579231948ffaf66cdda5439d3dc",
148
+
"choices": [
149
+
{
150
+
"finish_reason":"stop",
151
+
"index":0,
152
+
"logprobs":null,
153
+
"message": {
154
+
"content":"Arrr, I be Captain Chatbeard, the scurviest chatbot on the seven seas! Ye be wantin' to know me identity, eh? Well, matey, I be a swashbucklin' AI, here to provide ye with answers and swappin' tales, savvy?",
155
+
"role":"assistant",
156
+
"function_call":null,
157
+
"tool_calls":[],
158
+
"reasoning_content":null
159
+
},
160
+
"stop_reason":null
161
+
}
162
+
],
163
+
"created":1742496683,
164
+
"model":"Meta-Llama-3.1-8B-Instruct",
165
+
"object":"chat.completion",
166
+
"system_fingerprint":null,
167
+
"usage": {
168
+
"completion_tokens":66,
169
+
"prompt_tokens":32,
170
+
"total_tokens":98,
171
+
"prompt_tokens_details":null
172
+
},
173
+
"prompt_logprobs":null
174
+
}
175
+
```
149
176
**NOTE**: For multimodal models, currently only `ChatCompletion` is available, and only one image can be provided for each prompt.
Copy file name to clipboardExpand all lines: docs/source/index.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ user_guide
11
11
12
12
```
13
13
14
-
This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using [vLLM](https://docs.vllm.ai/en/latest/). **All scripts in this repository runs natively on the Vector Institute cluster environment**. To adapt to other environments, update [`launch_server.sh`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/launch_server.sh), [`vllm.slurm`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/vllm.slurm), [`multinode_vllm.slurm`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/multinode_vllm.slurm) and [`models.yaml`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/config/models.yaml) accordingly.
14
+
This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using [vLLM](https://docs.vllm.ai/en/latest/). **All scripts in this repository runs natively on the Vector Institute cluster environment**. To adapt to other environments, update the environment variables in [`cli/_helper.py`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/cli/_helper.py), [`cli/_config.py`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/cli/_config_.py), [`vllm.slurm`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/vllm.slurm), [`multinode_vllm.slurm`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/multinode_vllm.slurm), and model configurations in[`models.yaml`](https://github.com/VectorInstitute/vector-inference/blob/main/vec_inf/config/models.yaml) accordingly.
Copy file name to clipboardExpand all lines: docs/source/user_guide.md
+31-27Lines changed: 31 additions & 27 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,8 +4,7 @@
4
4
5
5
### `launch` command
6
6
7
-
The `launch` command allows users to deploy a model as a slurm job. If the job successfully launches, a URL endpoint is exposed for
8
-
the user to send requests for inference.
7
+
The `launch` command allows users to deploy a model as a slurm job. If the job successfully launches, a URL endpoint is exposed for the user to send requests for inference.
9
8
10
9
We will use the Llama 3.1 model as example, to launch an OpenAI compatible inference server for Meta-Llama-3.1-8B-Instruct, run:
11
10
@@ -18,8 +17,7 @@ You should see an output like the following:
18
17
19
18
#### Overrides
20
19
21
-
Models that are already supported by `vec-inf` would be launched using the [default parameters](vec_inf/config/models.yaml). You can override these values by providing additional parameters. Use `vec-inf launch --help` to see the full list of parameters that can be
22
-
overriden. For example, if `qos` is to be overriden:
20
+
Models that are already supported by `vec-inf` would be launched using the [default parameters](vec_inf/config/models.yaml). You can override these values by providing additional parameters. Use `vec-inf launch --help` to see the full list of parameters that can be overriden. For example, if `qos` is to be overriden:
You can also launch your own custom model as long as the model architecture is [supported by vLLM](https://docs.vllm.ai/en/stable/models/supported_models.html), and make sure to follow the instructions below:
31
-
* Your model weights directory naming convention should follow `$MODEL_FAMILY-$MODEL_VARIANT`.
29
+
* Your model weights directory naming convention should follow `$MODEL_FAMILY-$MODEL_VARIANT` ($MODEL_VARIANT is OPTIONAL).
32
30
* Your model weights directory should contain HuggingFace format weights.
33
-
* You should create a custom configuration file for your model and specify its path via setting the environment variable `VEC_INF_CONFIG`
34
-
Check the [default parameters](vec_inf/config/models.yaml) file for the format of the config file. All the parameters for the model
35
-
should be specified in that config file.
31
+
* You should specify your model configuration by:
32
+
* Creating a custom configuration file for your model and specify its path via setting the environment variable `VEC_INF_CONFIG`. Check the [default parameters](vec_inf/config/models.yaml) file for the format of the config file. All the parameters for the model should be specified in that config file.
33
+
* Using launch command options to specify your model setup.
36
34
* For other model launch parameters you can reference the default values for similar models using the [`list` command ](#list-command).
37
35
38
36
Here is an example to deploy a custom [Qwen2.5-7B-Instruct-1M](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-1M) model which is not
Alternatively, you can also use launch parameters to set these values instead of using a user-defined config.
66
+
Note that there are other parameters that can also be added to the config but not shown in this example, such as `data_type` and `log_dir`.
72
67
73
68
### `status` command
74
69
@@ -136,28 +131,37 @@ vec-inf list Meta-Llama-3.1-70B-Instruct
136
131
137
132
## Send inference requests
138
133
139
-
Once the inference server is ready, you can start sending in inference requests. We provide example scripts for sending inference requests in [`examples`](https://github.com/VectorInstitute/vector-inference/blob/main/examples) folder. Make sure to update the model server URL and the model weights location in the scripts. For example, you can run `python examples/inference/llm/completions.py`, and you should expect to see an output like the following:
134
+
Once the inference server is ready, you can start sending in inference requests. We provide example scripts for sending inference requests in [`examples`](https://github.com/VectorInstitute/vector-inference/blob/main/examples) folder. Make sure to update the model server URL and the model weights location in the scripts. For example, you can run `python examples/inference/llm/chat_completions.py`, and you should expect to see an output like the following:
140
135
141
136
```json
142
137
{
143
-
"id": "cmpl-c08d8946224747af9cce9f4d9f36ceb3",
144
-
"object": "text_completion",
145
-
"created": 1725394970,
146
-
"model": "Meta-Llama-3.1-8B-Instruct",
138
+
"id":"chatcmpl-387c2579231948ffaf66cdda5439d3dc",
147
139
"choices": [
148
140
{
149
-
"index": 0,
150
-
"text": " is a question that many people may wonder. The answer is, of course, Ottawa. But if",
151
-
"logprobs": null,
152
-
"finish_reason": "length",
153
-
"stop_reason": null
141
+
"finish_reason":"stop",
142
+
"index":0,
143
+
"logprobs":null,
144
+
"message": {
145
+
"content":"Arrr, I be Captain Chatbeard, the scurviest chatbot on the seven seas! Ye be wantin' to know me identity, eh? Well, matey, I be a swashbucklin' AI, here to provide ye with answers and swappin' tales, savvy?",
0 commit comments