Is there a way to turn-off tool calling json unicode escape for llama-server? #17517
Unanswered
TkskKurumi
asked this question in
Q&A
Replies: 1 comment
-
|
There is no server side option to turn this off. This is most likely a regression introduced in #16526 from setting |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I'm using llama-server to serve the model, python openai-API for client and Chinese prompting.
(server-launching command-line, client python source code are attached at the end.)
In the following screenshots, shows generation for calling get_weather(city="北京"), and continuos multi-turn to ask other city weather.
In the first round, model generates normal "北京", but its response to client is converted to "\uxxxx".
The first-round tool-message containing "\uxxxx" feed as context for the second round.
Causing model in-context-learn to generate "\uxxxx" for tool argument later. This can lead to catastrophic accuracy drop, since its much more difficult for model to generate unicode-hex than actual Chinese token.
Thank you very much for your time and assistance. Any insights or suggestions you might have would be greatly appreciated.
Attachments:
client.py
client-wa.py (work-around client)
serve0.log server logfile for the run.
Beta Was this translation helpful? Give feedback.
All reactions