GitBross Home Esplora Aiuto
Registrati Accedi
sol_87yf6cwp
/
LLMs-from-scratch
mirror da https://github.com/br8bit/LLMs-from-scratch.git
1
0
Forka 0
File Problemi 0 Wiki
Ramo (Branch): main
Rami (Branch) Tag
LLMs-from-scrat...
/
ch02
/
05_bpe-from-scratch
Matthew Hernandez 6f12edb0cc Fix issue: 731 by resolving semantic error (#738) 4 mesi fa
..
README.md a22d612be6 Bonus material: extending tokenizers (#496) 10 mesi fa
bpe-from-scratch.ipynb 6f12edb0cc Fix issue: 731 by resolving semantic error (#738) 4 mesi fa
tests.py 6f12edb0cc Fix issue: 731 by resolving semantic error (#738) 4 mesi fa

README.md

Byte Pair Encoding (BPE) Tokenizer From Scratch

  • bpe-from-scratch.ipynb contains optional (bonus) code that explains and shows how the BPE tokenizer works under the hood.