The AI industry pours engineering effort into GPU kernels and treats CPU preprocessing as an afterthought. Gigatoken shows how much performance that neglect can leave on the table.

In my first Gigatoken post, I tested its 1,000x claim against my tokenizer workload. I did not reproduce 1,000x, but the tokenizer core ran 30x to 40x faster, including 37.6x on a million-token request split into 1,024 segments. I wanted to understand where that gain came from.

I followed the path Gigatoken optimises most aggressively: tokenizers that apply BPE (byte pair encoding) to UTF-8 bytes, including GPT-2 and the tiktoken family. This is the path behind its headline throughput and the one exercised by my benchmark. The project has no technical paper yet, so what follows is my interpretation of Marcel Rød’s source comments, optimisation diary, profiling reports and commit history.1 Gigatoken describes SentencePiece as less optimised and does not support WordPiece.2

How BPE turns bytes into tokens

The tokenizers in this article turn UTF-8 text into a sequence of integers. Their vocabularies map byte sequences to token IDs. A token might represent a whole word, part of a word, punctuation or a single byte. It is an entry in the vocabulary, not necessarily a word.

Encoding is a pipeline rather than one long merge loop. Pretokenisation is the coarse split immediately before BPE: it divides the continuous byte stream into spans using model-specific rules, commonly expressed as a regular expression. Each intermediate span is a pretoken. It is input to BPE, not a final model token. For a GPT-2-style tokenizer:

" token 42!"  →  [" token", " 42", "!"]
How BPE turns bytes into token IDs UTF-8 bytes are split into independent pretokens. Each pretoken passes through repeated byte-pair merges before the remaining vocabulary entries become token IDs. UTF-8 bytes input text Pretokenise find model-defined spans BPE within each span [l] [o] [w] [e] [r] [lo] [w] [e] [r] Token IDs vocabulary entries Across pretokens: independent Within one: each merge changes the next choice
Pretokenisation divides the byte stream into model-defined spans. BPE may merge only within a span, so different spans can be processed independently. Within one span, each merge changes the candidates for the next merge.

The hard boundaries also explain why the spans are independent. BPE is forbidden to merge across them, so a merge inside one pretoken cannot create or remove a candidate pair in another. The encoder can process each pretoken separately, then concatenate their token IDs in the original order.

Inside one pretoken, BPE starts from bytes or initial symbols. Its merge table was learned when the tokenizer was trained and is fixed during encoding. Each legal adjacent pair has a numeric rank, and the lower rank wins. If lo is the best-ranked adjacent pair, the symbol sequence changes from [l] [o] [w] [e] [r] to [lo] [w] [e] [r]. The encoder looks up the affected neighbouring pairs again, chooses the next best merge and repeats until no legal merge remains. The surviving symbols map to token IDs.3

What Gigatoken changes inside the Rust core

Traditional Python tokenizer APIs inspect Python input objects and assemble Python-facing results; Gigatoken can read borrowed byte buffers and assemble a flat token buffer in Rust. The service I benchmarked already uses a Rust tokenizer core, so I exclude those API-boundary gains and examine the CPU and memory work inside tokenisation.4

I group the Rust-core changes into four families: 64-byte boundary classification, cached pretoken encodings, compact pair-rank tables for misses and parallel work split at safe pretoken boundaries. They cover the main hot path for the BPE tokenizers in scope but are not exhaustive or equally responsible for the speed-up. The measurements use different machines, cache states and controls, so each result should be read against its own baseline.

1. Pretokenise 64 bytes at a time

A general-purpose regex engine finds one matching span after another. Gigatoken’s mask scanner instead asks which positions in a block of 64 input bytes begin pretokens. The block is only a unit of computation, not a pretoken boundary.

The first classification pass handles ASCII, where one byte represents one character and simple byte comparisons can identify letters, digits, spaces, newlines, apostrophes and other categories. SIMD performs these comparisons across several byte positions at once. Each position is called a lane, and each lane holds one byte here. Four 16-byte NEON loads cover the batch on ARM, two 32-byte AVX2 loads cover it on x86, and AVX-512 can load all 64 bytes at once. The comparisons become separate 64-bit class masks, with one bit for each input byte. Bytes at or above 0x80 are marked for Unicode handling rather than assigned a character class in this pass.5

Gigatoken does not compile arbitrary regexes into this form. Its loader recognises a fixed set of tokenizer patterns and selects code written for that pattern. For each mask-scanner family, the Rust implementation expresses the regex’s boundary rules as operations over the class masks. The code treats each mask as a row of 64 on/off positions. A shift slides one row left or right so each byte lines up with its neighbour. AND keeps positions where two conditions are true, OR combines alternatives, and NOT selects positions outside a class. SIMD has finished once it creates the masks; ordinary 64-bit integer operations then combine them into boundary bits.

The author’s detailed optimisation notes and isolated scanner measurements use GPT-2’s r50k pretokenizer, so I use the same worked example here.5 Under r50k’s rules, a leading space can join the letter, number or punctuation run after it. In ·go·42!, where · is a space byte, shifting the letter and digit masks supplies the previous-position relationship for every byte at once. The resulting start bits identify ·go, ·42 and !.

The common 64-byte mask-scanner path A clean ASCII block passes through SIMD byte comparisons, 64-bit class masks and tokenizer-specific boundary rules. The resulting start-bit mask identifies the pretokens in input order. 64-byte block common ASCII path SIMD byte comparisons compare many positions as letters, digits, spaces and other classes 64-bit class masks one mask per class · bit i describes byte i Tokenizer-specific boundary rules in Rust shift masks to align neighbours · combine conditions with AND, OR and NOT Pretoken-start mask bytes · g o · 4 2 ! starts 1 0 0 1 0 0 1 Ordered pretokens: [·go] [·42] [!]
Here · is the space byte 0x20. The 64-byte block is a calculation width, not a semantic boundary: carry and lookahead allow a pretoken to continue across block edges.

Bit-parallel regex evaluation and SIMD block classification predate Gigatoken. Its contribution is specialising those techniques for the fixed patterns used by model tokenizers.6

The mask scanner must reproduce the regex’s boundaries. A pretoken can cross a 64-byte edge, so the scanner carries information from the preceding character and looks beyond the right edge where a rule needs it. Each batch returns the starts it can prove and marks any uncertain stretch as a bad zone for exact re-derivation. A bad zone is not necessarily malformed text; it means only that the block calculation cannot safely settle every boundary there.

Differential tests compare the combined mask and fallback output with the reference across crafted edge cases, 4,000 generated inputs and OpenWebText samples.5

Unicode stays exact

About 21% of batches in OpenWebText, a web-text corpus, contain at least one non-ASCII byte. UTF-8 breaks the ASCII pass’s one-byte-per-character assumption: one character can occupy several bytes, and those individual bytes do not reveal whether the character is a letter, number or whitespace. The SIMD pass therefore records their positions in a non-ASCII-byte mask for a second classification pass.

The second pass completes the same 64-position masks. For this r50k path, Gigatoken finds each UTF-8 lead byte, decodes its code point and uses a packed table of about 272 KiB to classify the character as a letter, number, whitespace or other. It stamps that class across every byte of the UTF-8 character, preventing a continuation byte from becoming a false boundary, then adds those results to the masks used by the tokenizer’s boundary rules.

Any region the masks cannot settle falls back to Gigatoken’s scalar walker. It uses ordinary integer instructions rather than SIMD lanes and advances from one span boundary to the next instead of classifying a whole 64-byte block. This fallback also handles CPUs without the required SIMD features, the incomplete tail of a buffer and bad zones such as invalid UTF-8 or ambiguous batch-edge cases. For r50k on the OpenWebText sample, only about 0.4% of batches require scalar re-derivation.7

How Unicode rejoins the mask path SIMD creates ASCII class masks and a mask of non-ASCII byte positions. If that second mask is non-empty, Gigatoken decodes UTF-8 code points, looks up their classes and stamps each class across the character's bytes. Updated masks feed the tokenizer's boundary rules. An ordered walker reads proven start bits and uses exact scalar advance only through uncertain gaps. SIMD byte classification ASCII class masks + non-ASCII-byte mask Any byte at or above 0x80? no Use the ASCII class masks nothing else to classify yes Extend the masks for Unicode 1. find UTF-8 leads and decode code points 2. look up each character's packed class 3. stamp that class across its UTF-8 bytes letter · number · whitespace · other Updated class masks Tokenizer-specific boundary rules operate over all 64 byte positions Batch result proven start bits + bad-zone bits One ordered boundary walker read proven start bits · scalar-advance exactly through any uncertain gap · emit pretoken spans
A non-ASCII byte updates the masks rather than automatically forcing fallback. The scalar walker re-derives boundaries only through bad zones, then continues the same ordered boundary stream.

When boundaries are requested one at a time, the scanner finds the lowest set bit with trailing_zeros and clears it with mask &= mask - 1.8 The hot encode path converts each mask into a flat buffer of boundary offsets, collects up to 256 spans, then processes them in a counted loop. This removes much of the data-dependent control flow from boundary discovery and consumption, but it does not remove the branches and dependent work in cache probes, BPE misses or token emission.

On GPT-2/r50k over a 1 GB OpenWebText sample, the mask scanner reached 2,460 to 2,600 MB/s against 983 MB/s for Gigatoken’s scalar reference, an isolated 2.5x to 2.6x improvement.5 The mask scanner makes pretoken boundaries cheaper to find, but the encoder must still turn each span into token IDs.

2. Cache the final IDs for each pretoken

After pretokenisation, an encoder must compute the token IDs for each span, normally by running the BPE merge loop. Gigatoken instead memoises the final token-ID sequence for the exact bytes of each ordinary pretoken. For an unseeded pretoken, the first occurrence computes the tokenizer’s normal answer; later occurrences copy the saved IDs. The cache records the answer for one pretoken, not its merge history or the completed tokenisation of a whole request.

Pretokens up to 15 bytes use a custom short-key table, while longer ones use a separate map. The short table is seeded with exact results for vocabulary byte strings from 1 to 15 bytes. A reused tokenizer instance keeps its results across calls and continues warming.

For a fixed tokenizer, the same pretoken bytes always produce the same IDs, so the cache can replay them without changing the result. Seed values use the tokenizer’s rules rather than assuming that a vocabulary entry’s own ID is always the right answer. The 128-bit key contains the complete short byte string and its length, and the table compares that full key after hashing. A hash collision can cause another probe but cannot return another pretoken’s tokens. Differential tests compare the cached path with uncached encoding across the supported tokenizer families.910

On the author’s 1 GB GPT-2/OpenWebText run, the table accumulated about 1.3 million unique short pretokens and served 99.4% of lookups as hits. About 90% of pretoken occurrences emitted one token and 98% emitted no more than two.10 In that workload, most spans skip the pair-rank lookups, merge decisions and scratch-state updates described in the next section. A hit only has to find the cached entry and copy its token IDs.

Fetch the key and answer together

Once caching removes the merge algorithm from the common path, the remaining cost is a largely random table lookup. Processors fetch memory in fixed blocks called cache lines. The x86 machines discussed here use 64-byte lines; the Apple M3 Pro in my benchmark uses 128-byte lines. L1 is the smallest and fastest cache near each core; L2 and the shared last-level cache hold more data at higher latency. A load that misses them may leave the core waiting for main memory.

A short pretoken and its length fit in one 128-bit key. A hash chooses an aligned home pair in the open-addressed table. Where available, Gigatoken calculates the hash with CPU checksum instructions; other targets use a portable arithmetic fallback. Each entry is 32 bytes, so both candidates form one 64-byte probe bucket that fits within a hardware cache line on both architectures. Gigatoken loads both keys and inline values together, compares the complete keys, and selects the match in registers. The common probe therefore requests the line containing the bucket instead of fetching metadata and then waiting for a dependent random load of the value.

Collisions probe later buckets. Up to four token IDs live inside an entry; larger answers spill into a separate append-only token buffer. Cache entries locate spilled answers by offset and length, so buffer reallocation does not invalidate the entries.

Gigatoken processes up to 256 pretokens as a group. While it discovers their spans, it asks the CPU to prefetch each future target line into L2. During the probe pass it requests promotion into L1 sixteen entries before use. A prefetch is a hint, not a guarantee. Its purpose is to overlap the memory fetch with independent work so the core is less likely to stall when the probe needs that line.

The common inline emit path also spends a few extra stores to remove control flow. It reserves room for four token IDs and writes all four lanes from the cache entry. The cursor advances only by the true count, so unused lanes are overwritten by the next result or truncated at the end. That avoids a count-dependent ladder of one-token, two-token, three-token and four-token branches.

Gigatoken's pretoken cache layout and prefetch ladder Two 32-byte entries form one 64-byte probe bucket. A 256-span pipeline first requests the future target line in L2, then requests it in L1 sixteen probes before it is consumed. One 64-byte probe bucket Entry 0 key u128 · 16 B inline IDs + spill ref up to 4 token IDs 16 B Entry 1 key u128 · 16 B inline IDs + spill ref up to 4 token IDs 16 B Memory-latency pipeline over 256 pretokens Discover span pack key + hash request target line in L2 16 probes ahead request line in L1 prefetch hint Probe + emit compare both keys write 1 to 4 IDs displaced hit, spill, long pretoken or miss → slow path
The key and the usual answer sit in the same 64-byte probe bucket, which fits within one hardware cache line. Software prefetches try to make that line resident before the probe needs it.

There is no published cache-on/cache-off A/B, so I would not assign memoisation alone a multiplier. In the campaign’s cold 10 GB GPT-2/OpenWebText benchmark, the combined probe-and-emit design, including staged prefetch, inline four-token values and flat output, was 27.6% faster in the single-threaded path that materialised the token IDs and 6.2% faster on the multithreaded path.9

Reduce address-translation work on Linux

The cache works on ordinary memory pages. On Linux, huge pages can reduce the address-translation overhead as a randomly probed table grows. Before a core can load a cache entry, it must translate the program’s virtual address into a physical address. The processor keeps recent translations in a translation lookaside buffer, or TLB. If the translation is absent, a page-table walk must find it first. On Zen, a software prefetch that misses the data TLB may be dropped, weakening the prefetch ladder described above.

A 64 MiB table occupies 16,384 ordinary 4 KiB pages but only 32 pages if it is fully backed by 2 MiB huge pages. Gigatoken therefore aligns a large short-cache allocation to 2 MiB on Linux and calls MADV_HUGEPAGE before the memory is first touched. If Linux honours the hint, fewer translations have to cover the same table.

In a separate Zen 5 whole-encode comparison, huge pages reduced warm page walks from about 28.6 million to about 2,300 per pass and improved warm throughput by 7.3%.11 That A/B also covered the input and output, so it measures address translation across the path rather than the cache table alone. The allocation hint is Linux-specific and does nothing on macOS.

3. Make pretoken-cache misses cheaper

A pretoken-cache hit skips BPE. On a miss, the encoder must run the merge loop for that pretoken. Each merge changes up to two neighbouring candidates, so the next choice depends on the previous one. Gigatoken keeps that ordering serial while shortening pair-rank lookups and reusing temporary storage.

Some BPE vocabularies assign merged token IDs in merge-priority order, allowing the ID to rank a candidate. Others store merge rank separately from token ID.12 Gigatoken preserves whichever ordering the tokenizer defines. For an ID-as-rank vocabulary that fits its packed representation, pairs whose IDs are both below 2,048 use a dense 16 MiB grid; other pairs use a packed sparse table. Both replace a general map with a shorter chain of dependent memory loads.13

Short and medium spans use fixed local rank and neighbour arrays with a linear scan. Long spans use reusable index arrays as a linked list plus a minimum heap, a priority queue that returns the lowest rank. On the ID-as-rank path, Gigatoken retains those arrays and the heap capacity between calls, avoiding fresh allocations for each cache miss. Both paths preserve merge priorities and choose the leftmost candidate when ranks tie.

On a 1 GB GPT-2 run on Zen 5, widening the dense grid to 16 MiB improved whole-encoder single-threaded throughput by 2.8% with a cold pretoken cache.13 Once warm, the cache’s 99.4% hit rate starves this path of work, and the gain disappears.10

4. Parallelise across proven boundaries

Hugging Face Tokenizers and tiktoken parallelise across caller-supplied inputs, so a million-token document remains one item. Gigatoken finds proven boundaries inside that document and assigns its chunks to several workers.14

Gigatoken cuts only at proven pretoken boundaries, which BPE cannot cross. It will not split added or special tokens; if no safe cut exists, the input stays serial. Tests compare the parallel output with the serial token IDs in their original order.14

Keep coordination outside the token loop

Workers share immutable vocabulary and pair-rank tables but keep private pretoken caches and scratch buffers. That keeps locks and cross-core traffic from shared writes out of the per-pretoken loop.

Workers claim chunks through an atomic counter. Because the chunks are arranged from largest to smallest, the largest are claimed first while smaller tail chunks keep cores busy near the end. Strict ordering prevents a large chunk from starting late and becoming the final straggler.14

Copy results while the tail is still encoding

Gigatoken reserves flat output space and uses a commit cursor to copy the ready prefix while later chunks are still encoding. This overlaps result copying and first-write page allocation with useful work. If the reservation is too small, it gathers the completed chunk buffers after encoding.14

Gigatoken's coarse parallel scheduling and output assembly A large input is cut at pretoken-safe boundaries into large early chunks and smaller tail chunks. Worker tasks pull chunks through an atomic index, use exclusive mutable state, then copy ready chunks into a flat output buffer in input order. One large input, cut only at safe boundaries large head chunks small tail large head chunks first, then the small tail shared model + atomic chunk index tasks pull one chunk at a time Worker task 0 exclusive cache + scratch chunk token buffer Worker task 1 exclusive cache + scratch chunk token buffer Worker task N exclusive cache + scratch chunk token buffer commit cursor copies chunks in input order
Each active task holds one mutable state slot exclusively. Shared work distribution and output assembly operate at chunk granularity.

In separate A/B comparisons, the 16-thread path was 6.2% faster with strict handout plus parallel gathering and 4.4% faster with opportunistic prefix copying, each against its own control.14

The closest published total comes from the campaign’s final same-session comparison on a cold 10 GB GPT-2/OpenWebText encode. Its documented single-thread path reached 1,039 MB/s, while the 16-thread path reached 8,792 MB/s. That is about 8.5x the wall throughput, equivalent to cutting the 10 GB encode from roughly 9.6 to 1.14 seconds. Each worker runs the scanner, cache and miss paths described above, so parallelism scales the faster core.14

What parallelism costs

Exclusive worker state removes shared-cache locking from the per-pretoken hot path, but it duplicates the cache and the work needed to warm it. In the author’s 16-worker profile, the state slots accumulated about 16 million distinct entries between them, compared with 5.5 million for a single cache. Aggregate multithreaded CPU time was 14.7 seconds against roughly 11 seconds for Gigatoken’s single-thread run, even though wall time fell sharply. The request finished sooner by spending more aggregate CPU work and memory.

Initial short-cache sizing is clamped between roughly 2 and 128 MiB per state slot, depending on the predicted share of the batch. The tables can continue to grow, however. The short cache has no eviction, and the long-key maps and token arenas are append-only. The pool keeps that memory across requests. One user processing several terabytes reports in an open issue that Gigatoken eventually consumed the RAM and swap of a 1 TB server.15

The wall-time gain therefore comes with a production requirement: long-running workloads with continually changing input need a way to bound or evict retained cache state.

What the Rust core achieved

Gigatoken’s 1,000x headline compares its native whole-buffer API with Hugging Face Tokenizers through its Python-facing batch API. Hugging Face’s encoder is itself multithreaded Rust, so this is not Python code against Rust code. The headline also includes the advantages of handing Gigatoken one 11.9 GB byte buffer, letting it find its own split points and avoiding compatibility work at the Python boundary.16

My service already used a multithreaded Rust tokenizer core, so my benchmark compared two Rust cores rather than a Python API with Gigatoken. As I reported in my first Gigatoken post, the timed million-token, 1,024-segment count path fell from about 159 milliseconds to 4.24 milliseconds, a 37.6x speed-up.

Together, the four mechanisms show how Gigatoken changes the way tokenisation runs on the hardware: classify 64 bytes at once, lay common cache probes out in 64-byte buckets, shorten the BPE miss path and parallelise at safe boundaries. GPU kernels receive this scrutiny as a matter of course. CPU preprocessing should too.

  1. This article follows the Gigatoken source revision used in my benchmark. Marcel Rød’s profiling campaign records the measurements and rejected experiments. ↩

  2. Gigatoken’s README distinguishes its heavily optimised BPE tokenizer path from SentencePiece and lists WordPiece as unsupported. ↩

  3. Byte pair encoding was introduced as a compression technique and adapted for subword tokenisation in Neural Machine Translation of Rare Words with Subword Units. OpenAI’s educational tiktoken implementation shows the repeated lowest-rank adjacent-pair merges during encoding. ↩

  4. The cleanest controlled measurement I found covers the input boundary only. Gigatoken replaced a corpus pre-split into per-document Python objects with borrowed byte buffers and separator splitting inside Rust. The author measured it 11% to 16% faster on 300 MB and the full 11.9 GB OpenWebText corpus, with identical token IDs. That does not isolate output materialisation or provide a full native-versus-compatibility multiplier. ↩

  5. The source documents the shared mask-scanner architecture, architecture-specific SIMD front ends and GPT-2 boundary algebra. The r50k module notes define the scalar path and report the isolated throughput. Separate mask implementations cover Qwen 2/3 and GPT-4o/o200k, while DeepSeek uses a specialised scalar walker. The r50k module also contains edge-case, fuzz and OpenWebText differential tests. ↩ ↩2 ↩3 ↩4

  6. Cameron et al.’s 2014 paper, Bitwise Data Parallelism in Regular Expression Matching, presents a general regex algorithm built from bitwise logic, shifts and one-bit-per-input-position streams. Langdale and Lemire’s simdjson paper describes a two-stage parser built around SIMD classification of input blocks; Gigatoken’s source calls its own movemask primitive simdjson-style. Gigatoken does not claim to have invented either underlying technique. ↩

  7. Gigatoken’s packed Unicode tables and Unicode mask fill preserve the fast mask representation. The r50k fallback handles ambiguous regions. ↩

  8. Here &= means “replace the value on the left with the result of a bitwise AND”. For a non-zero mask, subtracting one changes its lowest 1 bit to 0 and all the lower 0 bits to 1. ANDing that with the original clears only the lowest set bit: 1011000 & 1010111 = 1010000. trailing_zeros finds that bit’s position before it is cleared, so the next iteration can move to the following boundary. ↩

  9. See the campaign summary and method, its round-by-round results and the whole-encoder speculation profile. ↩ ↩2

  10. The cache module records the 1 GB OpenWebText distribution and layout rationale. The implementation documents exact cache seeding, key packing and hardware CRC hashing, paired cache probes and the staged prefetch loop. ↩ ↩2 ↩3

  11. The Zen 5 profile diagnoses the translation cost and measures the fix. The Linux kernel documentation explains transparent huge pages. ↩

  12. In an ID-as-rank vocabulary, merged token IDs follow merge priority. Other BPE vocabularies can assign the two independently. If b + c is rank 0 and produces ID 350 while a + b is rank 1 and produces ID 300, b + c must still merge first. Gigatoken’s loader checks whether merged IDs follow rank order, and the source includes a reversed-ID test. ↩

  13. The source explains the dense and sparse PairRankTable layout and the explicit-rank alternative. The Zen 5 profiling notes record the 16 MiB dense-grid A/B. ↩ ↩2

  14. Hugging Face’s encode_batch_fast parallelises over the input vector, one encode call per item. Gigatoken’s benchmark notes say that Hugging Face and tiktoken receive pre-split documents while Gigatoken receives the whole file and discovers its own split boundaries. The implementation documents safe split points, equivalence and added-token tests, tail-aware chunk sizing, strict work handout and opportunistic output commits. The parallel path is compared with serial output in unit tests and an ignored 1 GB OpenWebText test with added tokens. The profiling campaign measures 6.2% for strict handout plus parallel gather and 4.4% for opportunistic prefix copying. Its final same-session comparison reports 8,792 MB/s for the 16-thread ragged path and 1,039 MB/s for the single-thread materialising path on the same cold 10 GB campaign. These are different benchmark entry points and their 100 MB identity checks produce slightly different token counts, so 8.5x is a whole-path comparison rather than a strict token-identical scaling A/B. Gigatoken implements a token-identical serial ragged path, but does not publish a timing for it. ↩ ↩2 ↩3 ↩4 ↩5 ↩6

  15. The multithreaded profile quantifies duplicated cache work. The current cache has no eviction; issue #36 reports unbounded growth in a long-running terabyte-scale job. ↩

  16. Gigatoken’s README describes the headline comparison, notes that Hugging Face Tokenizers already runs multithreaded Rust and says Gigatoken’s compatibility mode does not reach 1,000x. The benchmark method gives Hugging Face pre-split Python strings and Gigatoken one unsplit byte buffer. ↩