Sebastian Raschka c21bfe4a23 Add PyPI package (#576)		8 kuukautta sitten
..
README.md	a22d612be6 Bonus material: extending tokenizers (#496)	10 kuukautta sitten
extend-tiktoken.ipynb	c21bfe4a23 Add PyPI package (#576)	8 kuukautta sitten

Extending the Tiktoken BPE Tokenizer with New Tokens

peilaus alkaen https://github.com/br8bit/LLMs-from-scratch.git

extend-tiktoken.ipynb contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via tiktoken and how to update the LLM accordingly