You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+3Lines changed: 3 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,6 +8,9 @@ GPT2 and GPT3 use byte pair encoding to turn text into a series of integers to f
8
8
9
9
This code was inspired by [Javascript implementation](https://github.com/latitudegames/GPT-3-Encoder) and partially generated by OpenAI himself!
10
10
11
+
> [!WARNING]
12
+
> This implementation of BPE tokenizer is not valid. See [https://x.com/burkov/status/1863007688470241336](https://x.com/burkov/status/1863007688470241336).
0 commit comments