- Author
- Anne Ouyang, Azalia Mirhoseini and Percy Liang, Stanford CRFM
- Published
- Undated (the page URL contains 2025/05/28)
- Kind
- Research group blog post with benchmark timings and code
- Original
- crfm.stanford.edu/2025/05/28/fast-kernels.html
- Subject
- Language models and training
- Themes
- Close to the machine, Handing over control
What it is
Anne Ouyang, Azalia Mirhoseini and Percy Liang report an early result. They set out to make synthetic data for training kernel models. Test-time search alone produced fast kernels, so they published early.
What it says
- AI-generated CUDA-C kernels matched or beat PyTorch's expert-tuned kernels on some tasks.
- Reported speeds on FP32 problems ranged from 101.3% for matmul to 484.4% for LayerNorm.
- Each round writes optimization ideas in plain language. Each idea spawns several kernel variants, and the best seed the next round.
- The post says sequential revision loops often stall on the same kinds of change.
- In the Conv2D example, speed rose from 20.1% of PyTorch to 179.9% over 13 rounds.
- Most best kernels appeared late, in round 4 or 5 of 5 rounds.
- Weaker results: FP16 matmul reached 52% and FP16 Flash Attention 9% of PyTorch.
What evidence it gives
The post gives timings for five kernels on one L40S GPU. It adds one step-by-step Conv2D trajectory. Correctness is checked on many random inputs within a 1e-02 tolerance. The full Conv2D code is in the appendix.
Interests and limits
The authors are Stanford researchers, supported by Standard Kernel Company and Prime Intellect. Results come from 10 KernelBench level 1 problems on one GPU, with modified sizes. The post says FP32 is less optimized, which may explain the wins. Gains hold for specific problem sizes. The setup allows lower-precision kernels within the 1e-02 tolerance. The authors call the search budget quite limited.