Sebastian Raschka c21bfe4a23 Add PyPI package (#576)		8 months ago
..
README.md	a22d612be6 Bonus material: extending tokenizers (#496)	10 months ago
extend-tiktoken.ipynb	c21bfe4a23 Add PyPI package (#576)	8 months ago

Extending the Tiktoken BPE Tokenizer with New Tokens

mirror of https://github.com/br8bit/LLMs-from-scratch.git

extend-tiktoken.ipynb contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via tiktoken and how to update the LLM accordingly