LLMs-from-scratch/ch05/09_extending-tokenizers

Extending the Tiktoken BPE Tokenizer with New Tokens

• extend-tiktoken.ipynb contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via tiktoken and how to update the LLM accordingly