GitBross Home Verkennen Help
Registreren Inloggen
sol_87yf6cwp
/
LLMs-from-scratch
spiegel van https://github.com/br8bit/LLMs-from-scratch.git
1
0
Vork 0
Bestanden Issues 0 Wiki
Aftakking: main
Aftakkingen Labels
LLMs-from-scrat...
/
ch02
/
05_bpe-from-scratch
Matthew Hernandez 6f12edb0cc Fix issue: 731 by resolving semantic error (#738) 4 maanden geleden
..
README.md a22d612be6 Bonus material: extending tokenizers (#496) 10 maanden geleden
bpe-from-scratch.ipynb 6f12edb0cc Fix issue: 731 by resolving semantic error (#738) 4 maanden geleden
tests.py 6f12edb0cc Fix issue: 731 by resolving semantic error (#738) 4 maanden geleden

README.md

Byte Pair Encoding (BPE) Tokenizer From Scratch

  • bpe-from-scratch.ipynb contains optional (bonus) code that explains and shows how the BPE tokenizer works under the hood.