- Author
- Kaya Omer (named in the citation block; the blog is called andthattoo)
- Published
- 2026-04-25
- Kind
- Personal blog post with two small benchmark runs
- Original
- andthattoo.dev/blog/structured_cot
- Subject
- Harnesses and agent loops
- Themes
- Saying it exactly, Small means
What it is
Kaya Omer tries a decoding grammar on a local Qwen3.6 model. The grammar limits the think block to labelled lines such as GOAL and APPROACH. The post compares this with free-form thinking and with a terse prompt.
What it says
- The author forced the think block of a Qwen3.6 model to follow a tiny grammar.
- On HumanEval+, the grammar cut thinking tokens about 22 times. Pass@1 was 92.7% against 92.1% for free-form thinking.
- A prompt-only terse control still spent 2298 thinking tokens on average.
- On 50 recent LiveCodeBench problems, pass@1 rose 14.0 points and thinking shrank 43.3 times.
- The author calls the gain partly mechanical, but not only empty-code recovery. Free-form runs often ended with empty code.
- On harder tasks, the model can move its deliberation into comments or post-think text.
- The author lists open limits: contamination, one model, one-shot coding tasks and grammar specificity.
What evidence it gives
The post gives results tables for HumanEval+ (164 problems) and a 50-problem LiveCodeBench v6 slice. It lists failure types for the free-form runs and a few per-task examples. It includes the grammar files.
Interests and limits
This is one author's blog post, not peer reviewed. The runs use one model and one quantization. The tasks are one-shot coding problems. The author warns that agentic tasks may behave differently.