- Author
- Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García and many co-authors, Nature Chemistry
- Published
- 20 May 2025
- Kind
- Peer-reviewed open-access paper introducing a benchmark, with a human expert baseline
- Original
- www.nature.com/articles/s41557-025-01815-x
- Subject
- AI put to use
What it is
A large author team led by Kevin Maik Jablonka's group presents ChemBench. It is an automated framework for testing chemical knowledge and reasoning in language models. The paper describes the question corpus, the evaluation method and a survey of expert chemists. It aims to measure ability and expose safety-relevant weaknesses.
What it says
- The authors built ChemBench: 2,788 questions, 1,039 written by hand and 1,749 semi-automatic.
- On ChemBench-Mini, o1-preview beat the best human by almost a factor of two.
- Models struggled on some tasks, such as predicting NMR signal counts, where o1-preview got 22%.
- Models were often no better than random at picking which of two molecules chemists prefer.
- Verbalized confidence was poorly calibrated, such as GPT-4 rating six wrong safety answers at 4.0.
- Retrieval over papers, as in PaperQA2, did not close the knowledge gaps in chemistry facts.
- The authors conclude teaching and exams need rethinking, since models beat the average human.
What evidence it gives
The paper offers scores for many models on 2,788 questions. Each was reviewed by at least two scientists. A smaller 236 question subset was answered by volunteers, with tool use allowed on part. It reports per-topic results, a preference test on 1,000 questions and a confidence analysis. Data and code are public.
Interests and limits
The authors made the benchmark and run its leaderboard. One author has been a paid contractor for OpenAI and others worked for Stability.AI. Only 19 experts took part. Of 16 who gave a background, 13 held master's degrees and were doing PhDs. Scoring was strict, and API refusals may have lowered some scores. Supplementary notes and figure images were not read.