Matthew Hernandez 6f12edb0cc Fix issue: 731 by resolving semantic error (#738)		4 月之前
..
README.md	a22d612be6 Bonus material: extending tokenizers (#496)	10 月之前
bpe-from-scratch.ipynb	6f12edb0cc Fix issue: 731 by resolving semantic error (#738)	4 月之前
tests.py	6f12edb0cc Fix issue: 731 by resolving semantic error (#738)	4 月之前

Byte Pair Encoding (BPE) Tokenizer From Scratch

镜像自地址 https://github.com/br8bit/LLMs-from-scratch.git

bpe-from-scratch.ipynb contains optional (bonus) code that explains and shows how the BPE tokenizer works under the hood.