- Author
- xnacly
- Published
- Jul 14, 2025
- Kind
- Personal blog post, a long tutorial in C with benchmarks
- Original
- xnacly.me/posts/2025/fast-lexer-strategies/
- Subject
- Compilers and interpreters
- Themes
- Close to the machine
What it is
xnacly is the author of purple-garden, an s-expression language built for personal use. The post explains the lexing stage and the optimizations applied to it. It warns that not every approach suits every use case.
What it says
- xnacly says a computed-goto jump table beats a switch. It works only with clang and gcc.
- A bump allocator behind an Allocator interface made one benchmark about 1.58x faster.
- Zero-copy string windows, instead of allocating copies, gave about a 1.4x speed-up.
- Hashing atoms while lexing lets keywords compare as numbers. Duplicate numbers are parsed once.
- Memory-mapping the input file made reading it 6x to 35x faster in the post's tests.
- The author reports lexing a million lines, 25,466,794 bytes, in 44ms on a laptop.
What evidence it gives
The post shows C code for each step. It quotes commit messages with timings, and some use hyperfine. It reports totals on a laptop and a tower. It compares with figures from another article.
Interests and limits
The author benchmarks own language on own synthetic input. The input repeats a fixed 15-line sample to reach about a million lines. Timings come from two machines and individual commits. The comparison table uses numbers from another article, so the setups may differ. The post says SIMD could go further but did not test it.
Connections
These links are this wiki's own reading, not claims the sources make.
- How To Make a Fast Dynamic Language Interpreter. Both lexer and interpreter gain by replacing strings with something cheap to compare. Atoms become hashed numbers in one, and Symbol objects in the other. The trick carries because a name's identity matters more than its spelling.
- I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code. The 70x parser figure came from a small test corpus, and production looked different. xnacly's numbers also rest on the author's own synthetic input. A speed claim needs a stated workload. Both sources let you see theirs, which is the useful habit.