Sebastian Raschka c21bfe4a23 Add PyPI package (#576)		8 mēneši atpakaļ
..
README.md	a22d612be6 Bonus material: extending tokenizers (#496)	10 mēneši atpakaļ
extend-tiktoken.ipynb	c21bfe4a23 Add PyPI package (#576)	8 mēneši atpakaļ

Extending the Tiktoken BPE Tokenizer with New Tokens

spogulis no https://github.com/br8bit/LLMs-from-scratch.git

extend-tiktoken.ipynb contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via tiktoken and how to update the LLM accordingly