Skip to content

Commit 34d2854

Browse files
authored
Update README.md
1 parent 27aaadf commit 34d2854

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,9 @@ GPT2 and GPT3 use byte pair encoding to turn text into a series of integers to f
88

99
This code was inspired by [Javascript implementation](https://github.com/latitudegames/GPT-3-Encoder) and partially generated by OpenAI himself!
1010

11+
> [!WARNING]
12+
> This implementation of BPE tokenizer is not valid. See [https://x.com/burkov/status/1863007688470241336](https://x.com/burkov/status/1863007688470241336).
13+
1114
## Install
1215

1316
```bash

0 commit comments

Comments
 (0)