So far, I haven’t found any major issues at all:
I went through the guide mostly looking for places where the optimization steps could accidentally change tokenizer semantics rather than just make the same algorithm faster. From the public guide and the small checks I could reproduce, I did not find evidence of a major problem.
The overall progression also makes sense to me:
naive pair recounting → incremental counts → parallel pretokenization → pair index → lazy heap → larger multilingual training → .tiktoken export
That is a nice progression because each optimization removes one concrete source of wasted work instead of jumping straight from a pedagogical implementation to an opaque optimized one. It also lines up reasonably well with the kinds of data structures used in mature BPE trainers; for example, the current Hugging Face BPE trainer also maintains pair counts, tracks where pairs occur, uses a priority queue, and re-checks stale candidates. There is also a useful theoretical treatment in A Formal Perspective on Byte-Pair Encoding, which derives a faster BPE trainer while treating the learned merge sequence itself as the object that must be preserved.
If I added only one thing to this tutorial, though, it would be a small exact merge-sequence regression test across the optimization stages.
The reason is that the current performance section intentionally has no automated evaluation, and the manual checks mostly compare runtime plus the final token count/compression:
naive
↓
incremental pair counts
↓
pair index
↓
heap
At each arrow, I think it would be useful to preserve one stronger invariant:
assert incremental_merges == naive_merges
assert indexed_merges == naive_merges
assert heap_merges == naive_merges
Optionally, a few fixed input strings could also assert identical encoded token IDs.
This would fit especially well with something you already do in the parallel-pretokenization chapter: you compute the sequential and parallel pretoken-frequency dictionaries and require exact equality before benchmarking. I think that is a very good pattern. Extending the same idea to the later optimization stages would make it explicit that each speedup changes the data structure/work performed, not the BPE that is learned.
Why I think merge-list equality is worth checking
The baseline specification is already deterministic. In the BPE implementation chapter, equal-frequency pairs are broken by comparing the bytes represented by the left tokens and then the right tokens, and encoding later respects the order in which merges were learned.
That makes the ordered merge list part of the tokenizer’s semantics, not just an implementation detail.
A small independent sanity check made the distinction fairly visible. I exhaustively tried 9,837 strings over a tiny A/B/C alphabet (lengths 2–8, up to four merges), comparing:
- the documented represented-bytes tie ordering, and
- a deliberately different tie ordering based on token IDs.
The merge histories differed for 4,676 / 9,837 strings.
More interestingly, all 4,676 divergent cases still finished with the same final token count in that small experiment.
The smallest example I captured was:
BAAA
After the first AA merge, two one-count candidates can remain. Ordering them by the represented bytes versus ordering them by the newly minted numeric IDs can choose a different next merge. Both paths can still end at the same token count, so a token-count/compression check alone does not necessarily expose the semantic difference.
This is not evidence that your heap implementation uses the wrong tie key — the optimized learner implementation is not public here, so I cannot make that claim. It is just a small example of the type of regression that exact merge-list equality would catch.
A few very cheap fixtures could cover most of the awkward shapes:
aaa
aaaa
aaaaa
abababa
BAAA
plus perhaps one Unicode example after UTF-8 conversion and one case constructed specifically to create several equal-frequency winners.
If a regression ever fails, the debugging output can also stay tiny. I would stop at the first divergent round and print only something like:
round
winner pair
winner frequency
bytes represented by left/right token
new rank / token ID
That should usually localize whether the divergence came from pair-count maintenance, the pair index, stale heap entries, or tie ordering.
There are real-world reasons to treat this as a useful invariant rather than just a theoretical concern. For example, Hugging Face has had separate BPE trainer issues where pair-count overflow changed merge ordering and where count/heap bookkeeping allowed a zero-occurrence pair to become a winner. Those are different implementations and different failure mechanisms, so I would not infer that either bug exists here. They are just examples of why a trainer can apparently keep running while the learned merge sequence is no longer the intended one.
The current Hugging Face trainer’s own tests are also a useful reference point: they assert not only the resulting vocabulary but the expected merge pairs and their ranks directly in trainer.rs.
A few very small documentation-level things
None of these look like architectural problems; they seem more like inexpensive polish.
1. The pair-index pseudocode could probably snapshot the affected IDs explicitly.
The prose in the pair-index chapter says:
copy its affected IDs before modifying the index
but the pseudocode immediately below currently shows:
affected_ids = pair_index[winner]
for pretoken_id in affected_ids:
remove the pretoken's old pairs from the counts and index
apply the winning merge
add the pretoken's new pairs to the counts and index
If removing the old pairs can mutate the same set stored at pair_index[winner], a literal Python implementation can hit:
RuntimeError: Set changed size during iteration
So if the intended implementation already snapshots it, showing that directly in the pseudocode would make the prose and code agree:
affected_ids = pair_index[winner].copy()
or equivalently:
affected_ids = list(pair_index[winner])
This may be purely a documentation issue rather than an implementation issue.
2. It may help to surface the Python 3.14 requirement earlier.
The heap chapter correctly says that it is using Python 3.14’s max-heap API:
heapify_max(...)
heappush_max(...)
heappop_max(...)
Those functions were indeed added in Python 3.14.
The only practical wrinkle is that a reader can get quite far through the earlier setup with an older Python. For example, the current free Colab runtime I checked was Python 3.13.15, where those public max-heap functions are not available.
So either of these would probably avoid some reader friction:
- make
Python >= 3.14 visible near the initial environment/setup instructions, or
- mention the conventional min-heap-with-reversed-priority fallback for Python <=3.13.
The heap chapter itself is already correct; this is mostly about making the dependency visible before someone reaches that chapter.
3. I think 11 kB is just a typo.
The scale-up chapter currently says:
Tiny Shakespeare corpus of 11 kB
whereas the earlier parallel chapter calls it roughly 1.1 MB.
I fetched the same public Tiny Shakespeare file used by the tutorial and got:
1,115,394 bytes
≈ 1.064 MiB
so ~1.1 MB looks like the intended value.
One part I especially liked: the parallel-pretokenization boundary check
I spent a little more time on this because chunk boundaries are an easy place for a tokenizer implementation to become subtly different.
Your rule — split only immediately before a whitespace run, so the regex boundary is preserved — looks sensible, and the important part is that you do not merely assume it works: the guide tells the reader to compare the entire sequential and parallel frequency dictionaries exactly.
I ran a small property-style check using the same GPT-2 regex on 5,000 seeded generated strings containing English, several other scripts, punctuation, spaces, tabs, CR/LF combinations, and emoji.
For the documented whitespace-boundary rule:
Counter mismatches: 0 / 5000
For deliberately arbitrary near-equal chunk boundaries:
Counter mismatches: 4550 / 5000
Finite generated testing obviously is not a proof, but it pushed me in the direction of more confidence in this part of the guide, not toward a bug report. It also reinforces why I think the same exact-equality style would be valuable for the later BPE optimization steps.
A couple of optional directions I would keep optional
There are other interesting tokenizer questions around this project, but I would not make them blockers for this tutorial.
Per-language token cost
Because the later example becomes multilingual, a small held-out table of bytes/token by language could be a useful diagnostic. It could show concretely how the global BPE merge budget gets distributed across scripts/languages.
I would avoid presenting this as a tokenizer-quality score, though. Compression/token efficiency is only one property of a tokenizer, and downstream model quality does not reduce to minimum token count.
This also connects nicely to the recent work on multilingual allocation such as Parity-Aware Byte-Pair Encoding, but I think your current “where to go next” framing is already the right place for that kind of topic. I would not complicate the core classical-BPE walkthrough with it.
Unicode normalization
Byte-level BPE gives complete byte coverage, but that is conceptually separate from Unicode normalization. For example, canonically equivalent Unicode strings can still have different UTF-8 byte sequences.
I do not think that means this tutorial needs to add normalization. “Raw UTF-8 bytes with this pretokenization regex” is a perfectly coherent contract. A short note making that boundary explicit might help a future reader, but changing normalization would change what the tokenizer is being trained to model and would be a separate design choice.
Production special-token behavior
Similarly, special-token policy, normalization, runtime safety checks, and all of the other things a production tokenizer may need do not have to be pulled into a from-scratch BPE tutorial.
The important boundary is just that a serialized rank table is not necessarily the whole tokenizer contract: the pretokenization regex and special-token policy also affect runtime behavior. Your .tiktoken section already moves in the right direction by checking token-ID parity rather than merely checking that a file can be written and loaded.
So, if I were prioritizing this purely by information gained per amount of extra tutorial/code work, I would probably do:
- Exact ordered-merge regression across the optimization stages — the main addition.
- Make the pair-index pseudocode snapshot explicit with
.copy() — tiny documentation fix.
- Surface Python 3.14 earlier or mention a <=3.13 fallback — practical reader fix.
- Fix
11 kB → ~1.1 MB — typo.
- Leave the broader multilingual/normalization/production-tokenizer topics as optional follow-ups.
The nice part is that none of this requires changing the tutorial’s direction. The naive implementation is already a very useful reference oracle, and the sequential-vs-parallel chapter already demonstrates the right testing idea. I would mostly reuse those two strengths to make the optimization chapters a little more self-checking.