- Author
- Posted by jasondavies, with replies from many commenters
- Published
- 16 July 2026
- Kind
- Hacker News discussion thread on a project page claiming a benchmark result
- Original
- news.ycombinator.com/item?id=48935905
- Subject
- Harnesses and agent loops
What it is
The linked page is schema-harness.github.io, which announces a harness called Schema. The thread has 131 points and was posted by jasondavies. Replies argue about the public set, the harness method and benchmark design.
What it says
- Commenter westurner quotes the page: 99% with Opus 4.8 and Fable 5. It quotes 95.35% with GPT-5.6 Sol.
- Commenter modeless says the public set is known in advance. Only the private score matters.
- Commenter Tadpole9181 says the AI builds its own copy of the game. That bypasses the benchmark's intent.
- Commenter snewman replies that writing a simulator requires understanding the game.
- Commenter ubermon quotes a rule that reruns weak games with stronger models.
- Commenter gandalfgeek urges caution because the harness is not open-sourced or tested on held-out data.
- Commenter UltraSane says GPT 5.6 Sol cost $25,000 to run on ARC-AGI-3.
What evidence it gives
The thread has the project's quoted claim and a quoted scoring rule, plus opinion. Commenters cite a leaderboard and a related paper in a side debate. No commenter reports reproducing the result.
Interests and limits
The project page was not read. Its claims reach this page only as quotes from commenters. The scores are on the public set, not the held-out set. The harness was not open-sourced, per commenters. The $25,000 cost figure is one unsourced comment.