Skip to content

feat: add suport for mistral3 models #524

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
albertkawmi wants to merge 3 commits into from
Closed

feat: add suport for mistral3 models #524

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
305bff6
feat: add suport for `mistral3` models
albertkawmi Dec 3, 2025
7af234f
feat: resolveChatWrapper for mistral3
albertkawmi Dec 4, 2025
d9e8fcd
Revert "feat: resolveChatWrapper for mistral3"
albertkawmi Dec 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/chatWrappers/utils/resolveChatWrapper.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,7 @@ export function resolveChatWrapper(
return createSpecializedChatWrapper(Llama3_1ChatWrapper);
else if (includesText(modelNames, ["llama 3", "llama-3", "llama3"]))
return createSpecializedChatWrapper(Llama3ChatWrapper);
else if (includesText(modelNames, ["Mistral", "Mistral Large", "Mistral Large Instruct", "Mistral-Large", "Codestral"]))
else if (includesText(modelNames, ["Mistral", "Mistral Large", "Mistral Large Instruct", "Mistral-Large", "Mistral 3", "mistral3", "Ministral", "Codestral"]))
Copy link
Member

@giladgd giladgd Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
else if (includesText(modelNames, ["Mistral", "Mistral Large", "Mistral Large Instruct", "Mistral-Large", "Mistral 3", "mistral3", "Ministral", "Codestral"]))
else if (includesText(modelNames, ["Mistral", "Mistral Large", "Mistral Large Instruct", "Mistral-Large", "Codestral", "Ministral"]))

The first text "Mistral" already includes the "Mistral 3", "mistral3" variations (since the match is case-insensitive)

return createSpecializedChatWrapper(MistralChatWrapper);
else if (includesText(modelNames, ["Gemma", "Gemma 2"]))
return createSpecializedChatWrapper(GemmaChatWrapper);
Expand Down Expand Up @@ -454,6 +454,8 @@ export function resolveChatWrapper(
return createSpecializedChatWrapper(FalconChatWrapper);
else if (arch === "gemma" || arch === "gemma2")
return createSpecializedChatWrapper(GemmaChatWrapper);
else if (arch === "mistral3")
return createSpecializedChatWrapper(MistralChatWrapper);
}

return null;
Expand Down
1 change: 1 addition & 0 deletions src/gguf/types/GgufMetadataTypes.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ export const enum GgufArchitectureType {
grovemoe = "grovemoe",
apertus = "apertus",
cogvlm = "cogvlm",
mistral3 = "mistral3",
clip = "clip",
unknown = "(unknown)"
}
Expand Down
30 changes: 30 additions & 0 deletions test/standalone/chatWrappers/utils/resolveChatWrapper.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -767,4 +767,34 @@ describe("resolveChatWrapper", () => {
});
expect(chatWrapper).to.be.instanceof(HarmonyChatWrapper);
});

test("should resolve to MistralChatWrapper based on mistral3 architecture", () => {
const chatWrapper = resolveChatWrapper({
fileInfo: {
version: 3,
tensorCount: 0,
metadata: {
general: {
architecture: "mistral3",
// eslint-disable-next-line camelcase
quantization_version: "1"
},
tokenizer: {
ggml: {
model: "llama",
tokens: [],
// eslint-disable-next-line camelcase
token_type: []
}
}
} as any,
metadataSize: 0,
architectureMetadata: {} as any,
splicedParts: 1,
totalTensorCount: 0,
totalMetadataSize: 0
}
});
Copy link
Author

@albertkawmi albertkawmi Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can tidy up this object stubbing... but I'm more wondering if this test case is adding value? There aren't similar test cases for model architecture in this file. Can delete this if you prefer.

Copy link
Member

@giladgd giladgd Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

node-llama-cpp attempts to compare the behavior of a model's Jinja template to one of the builtin chat wrappers so that it can use the chat wrapper instead, which then can further enhance the original chat template by either introducing/enhancing function calling support, customizing the chat template building and parsing (for example, to support gpt-oss's comment segments), or allowing further customization using one of the constructor parameters.
The tests in this file are used to ensure that the Jinja templates of popular models are detected and mapped correctly to the relevant builtin chat wrappers.

I've just tested the hf:mistralai/Ministral-3-3B-Instruct-2512-GGUF:Q4_K_M model and it seems that theJinjaTemplateChatWrapper chat wrapper was able to correctly parse the model's chat template and detect the function calling syntax, so that it worked out of the box.

Sometimes, some models come with Jinja templates that use features that we don't support yet, so until I or some else implements those missing features, the fallbacks in the resolveChatWrapper method are used to pick up on other cues from the model's metadata in an attempt to match it with the right chat wrapper so that the model can still be used.

The Mistral 3 chat template seems different than previous Mistral models, so the current builtin Mistral chat wrapper is incompatible with it and thus wasn't picked up by default (you can see the Wrapper: JinjaTemplate in the screenshot you shared).

If you'd like, you can create a Mistral3ChatWrapper class to support the new chat template and then add a test case for it here to ensure the Jinja template of the model automatically maps to the Mistral3ChatWrapper.
I haven't found any enhancement/customization opportunities in Mistral 3's chat template, so I think the JinjaTemplateChatWrapper chat wrapper support suffices for it for now, but if you'd like to implement Mistral3ChatWrapper then you can definitely do that.

The test you added here indeed doesn't add value, so you can remove it.

expect(chatWrapper).to.be.instanceof(MistralChatWrapper);
});
});
Loading