Sebastian Raschka c21bfe4a23 Add PyPI package (#576)		8 月之前
..
README.md	a22d612be6 Bonus material: extending tokenizers (#496)	10 月之前
extend-tiktoken.ipynb	c21bfe4a23 Add PyPI package (#576)	8 月之前

Extending the Tiktoken BPE Tokenizer with New Tokens

镜像来自 https://github.com/br8bit/LLMs-from-scratch.git

extend-tiktoken.ipynb contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via tiktoken and how to update the LLM accordingly