Recommended reading.
"Gigatoken can tokenize the entire internet in <；7 hours on one machine."
It can tokenize data at GB/s. That's a big deal.
>； pip install gigatoken
Introducing the world's fastest tokenizer implementation, Gigatoken! Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken fo...