- Author
- Timothy B. Lee, Understanding AI
- Published
- 23 June 2025
- Kind
- Newsletter explainer drawing on cited papers and one interview
- Original
- www.understandingai.org/p/reinforcement-learning-explained
- Subject
- Language models and training
- Themes
- Getting good, Feedback loops
What it is
Timothy B. Lee writes a plain-language explainer for the newsletter Understanding AI. It traces how reinforcement learning, after pretraining, helped make agentic models possible. It opens a week-long series on agents.
What it says
- The author thinks AutoGPT and BabyAGI failed because small errors compounded over many steps.
- Ross and Bagnell argued in 2011 that imitation learning suffers compounding errors.
- In DAgger, the model drove and Ross corrected it, which worked much better.
- Reinforcement learning trains by trial and error. A 2025 paper found it held up better out of distribution.
- RLHF trains a text-rating model from human choices, then uses it as the reward model.
- Constitutional AI has a judge model score outputs against written principles.
- DeepSeek's R1 report showed reasoning length growing during training on checkable math and code problems.
What evidence it gives
The article draws on cited papers and a quoted interview with Ross. It walks through Ross's SuperTuxKart experiment. It describes a 2025 DeepMind chart, a 2022 Waymo paper and DeepSeek's R1 paper. It runs no experiments of its own.
Interests and limits
The author runs a paid newsletter, and the article promotes a subscription discount. The title promises a minimum of math and jargon. The Anthropic Opus account rests on a Semianalysis report. The author says DeepSeek's discovery was not entirely spontaneous.
Connections
These links are this wiki's own reading, not claims the sources make.
- The quality of AI-assisted software depends on unit of work management. Raykar computes it: a 5% error rate over 10 turns gives about 59.9% success. Lee traces the same worry to a 2011 paper on imitation learning. There, correcting the model while it drove worked much better. Is a review loop a kind of DAgger?