- Author
- Atharva Raykar (nilenso)
- Published
- 15 September 2025
- Kind
- Company blog opinion post, with napkin arithmetic
- Original
- blog.nilenso.com/blog/2025/09/15/ai-unit-of-work/
- Subject
- Harnesses and agent loops
- Themes
- Handing over control, Checking the work
What it is
Atharva Raykar writes for the nilenso blog about getting better results from AI coding agents. The post links context size, error compounding and task size. It ends by proposing user stories plus extra context as the unit.
What it says
- Raykar says the main bottleneck in AI-assisted coding is correct context, not intelligence.
- Errors compound over turns: a 5% error rate over 10 turns gives about 59.9% success.
- METR's tasks are less messy than real software work. The author rates that work at least around 7 to 8 of 16.
- The author extrapolates that GPT-5 falls from 70% to around 40% on two-hour tasks.
- A right-sized unit of work is small, concise and legible to humans.
- User stories are a good starting unit, though the author says they need something more.
- nilenso's StoryMachine experiment reads a PRD and tech specs and produces story cards.
What evidence it gives
The post gives a table of compounded success rates from assumed per-step error rates. It reads one METR paper and extrapolates from its messiness measure. It adds the author's own experience.
Interests and limits
nilenso is a consultancy that is running the StoryMachine experiment mentioned in the post. The per-step error rates are assumed, not measured. The extrapolation from METR is the author's own, and the proposed unit is untested.
Connections
These links are this wiki's own reading, not claims the sources make.
- Reinforcement learning, explained with a minimum of math and jargon. Raykar computes it: a 5% error rate over 10 turns gives about 59.9% success. Lee traces the same worry to a 2011 paper on imitation learning. There, correcting the model while it drove worked much better. Is a review loop a kind of DAgger?