Commit b70fd3a
committed
server : use common_token_to_piece instead of common_detokenize
This commit replaces the call to common_detokenize with
common_token_to_piece in the populate_token_probs.
The motivation for this change is to avoid an issue where
common_detokenize would remove the word boundary character for tokens,
which caused a regression in the server generated token probabilities.
Resolves: #117281 parent d7b31a9 commit b70fd3a
1 file changed
+1
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2297 | 2297 | | |
2298 | 2298 | | |
2299 | 2299 | | |
2300 | | - | |
| 2300 | + | |
2301 | 2301 | | |
2302 | 2302 | | |
2303 | 2303 | | |
| |||
0 commit comments