Tokenization is the one part of the language modeling stack that almost nobody profiles. Gigatoken, released by Marcel Rød (a PhD student from Stanford) under an MIT license, argues that this was a mistake. The library encodes text at gigabytes per second on a single machine, against baselines that are already multithreaded Rust.
The GPT-2 tokenizer benchmarking yields remarkable results: evaluated on the 11.9 GB owt_train.txt corpus using a 144-core AMD EPYC 9565 dual-socket setup, Gigatoken processes data at a staggering 24.53 GB/s. In comparison, OpenAI’s tiktoken achieves 36.0 MB/s, while HuggingFace tokenizers registers at 24.8 MB/s on the identical hardware configuration. These marks demonstrate performance advantages of 681x and 989x, respectively.
On an Apple M4 Max with 16 cores, the same GPT-2 workload runs at 8.79 GB/s, or 1,268x HuggingFace tokenizers and 140x tiktoken. On a consumer AMD Ryzen 7 9800X3D, it runs at 6.27 GB/s, or 106x and 68x. The speedup is not an artifact of one CPU or one vocabulary.
What is Gigatoken
Gigatoken is a byte-pair encoding (BPE) tokenizer written in Rust with Python bindings. It ships on PyPI as gigatoken (version 0.9.0, released 21 July 2026) and installs with pip install gigatoken. The repository is 66.2% Rust and 33.3% Python. It supports 23 distinct tokenizer families in the published benchmarks, covering GPT-2, GPT-OSS, Llama 3 through 4, Qwen 2 through 3.6, DeepSeek V3/R1/V4, GLM 4 and 5, Kimi K2, Nemotron 3, Phi-4, OLMo 2/3, ModernBERT, Gemma and Mistral.
There are two ways to use it. Compatibility mode wraps an existing HuggingFace or tiktoken tokenizer and preserves exact output parity, at a real cost to throughput. The author (Marcel) states on Hacker News that compatibility mode delivers roughly 200–300x depending on usage, because it still pays Python overhead for list creation and string-to-bytes conversion. The native Gigatoken API lets Rust read files directly and is where the published numbers come from.
The interactive benchmark explorer
Every number below is drawn from the repository’s benchmarks section and the pretokenizer optimization log. Switch CPUs, walk the optimization history, or estimate how long your own corpus would take.
How it gets there
The gains do not come from a better BPE merge loop. They come from two places that most tokenizers treat as solved.
(1) Pretokenization: Most implementations delegate this to a regex engine. Gigatoken hand-writes it. The optimization log tracks single-threaded GPT-2 pretokenizer throughput on 100 MB of OpenWebText, and the progression is instructive:
- A fancy-regex baseline runs at ~47 MiB/s. A hand-rolled state machine hits ~380 MiB/s. A winnow-combinator implementation with NEON SIMD intrinsics reaches 462 MiB/s.
- Replacing winnow with a direct Iterator, adding a 256-byte class lookup table for O(1) first-byte dispatch, and switching from NEON intrinsics to SWAR (SIMD Within A Register) pushes it to 830 MiB/s. SWAR loads 8 bytes as a u64 and checks all 8 for the letter property with branchless arithmetic. It needs no architecture-specific intrinsics.
- The final step is dual-cursor ILP exploitation, reaching 1,049 MiB/s. The insight is that the bottleneck at ~840 MiB/s was latency, not throughput. Each token’s end position depends on the previous one, a serial chain of roughly 25–27 cycles. Running two independent cursors from a safe split point lets the out-of-order engine interleave both streams across idle execution ports.
Net effect on the pretokenizer alone: 2.27x over the winnow + NEON baseline, and 22.3x over the regex implementation.
(2) Pretoken caching: If a word has been seen before, its encoded tokens are looked up rather than recomputed. The author (Marcel) notes this is hard in practice, because the cache grows quickly and pretoken distributions are long-tailed. On top of that, interactions with Python are minimized and threads are designed to interact minimally with each other.
The optimization log is unusually honest about what failed. A hot/cold split using #[cold] and #[inline(never)] regressed to 580 MiB/s and was reverted, because the inline barrier stopped LLVM from optimizing the combined ASCII and unicode loop. A two-pass classification buffer with SWAR transition counting was algorithmically correct but ran at 354 MiB/s, since the extra memory traffic beat the branch savings. Profile-guided optimization had no measurable effect, because the inner loop is already branchless and the word-boundary branch is data-dependent.
Benchmark Methodology Notes
The comparison is not strictly apples-to-apples. Gigatoken encodes entire un-split files, finding its own document boundaries and parallelizing automatically. In contrast, HuggingFace (encode_batch_fast) is evaluated on the first 100 MB and tiktoken (encode_ordinary_batch) on the first 1 GB, both pre-split on <|endoftext|>. Because the baselines omit caching, their throughput remains uniform. Measurements report the best of three interleaved rounds using fresh processes with parallelism enabled.
Vocabulary types also introduce constraints; SentencePiece tokenizers are only partially optimized. On EPYC, Gemma 1 processes at 2.51 GB/s (7.3x speedup), Gemma 3 at 3.43 GB/s (9.6x), and CodeLlama at 3.47 GB/s (10.0x). Although substantial, these gains are an order of magnitude lower than the headline BPE performance.
An independent reproduction on KrabArena verified the results. On a 4-vCPU Intel Xeon VM (2.20 GHz) with a 174 MB OpenWebText slice, Gigatoken 0.9.0 achieved a median of 277.8 MB/s, outperforming tiktoken 0.13.0 (10.62 MB/s) by 26.2x and tokenizers 0.23.1 (3.33 MB/s) by 83.4x. All trials successfully validated 35,356 documents, confirming the performance trend scales with core count.
Key Takeaways
- Gigatoken hits 24.53 GB/s on GPT-2 on a 144-core EPYC, 989x HuggingFace tokenizers and 681x tiktoken.
- The speedup holds across x86 and ARM, and across 23 tokenizer families, not one tuned configuration.
- Gains come from a hand-written SWAR pretokenizer and pretoken caching, not a faster BPE merge loop.
- SentencePiece vocabularies see 7–22x, not 1,000x, and WordPiece is unsupported.
- Compatibility mode preserves exact HuggingFace output parity at roughly 200–300x instead of the full speedup.
Check out the GitHub repo and the release thread. All credit for this research goes to the author of this project.