- Author
- Posted by zambelli (Antoine Zambelli, AI Director at Texas Instruments), with replies from many commenters
- Published
- 19 May 2026
- Kind
- Show HN thread where the author presents an open-source tool and answers questions
- Original
- news.ycombinator.com/item?id=48192383
- Subject
- Harnesses and agent loops
What it is
Antoine Zambelli posted Forge as an open-source layer around self-hosted models. It retries and nudges models when tool calls fail. The post cites a peer-reviewed paper, a repo, a dashboard and a demo video.
What it says
- Zambelli claims Forge lifts an 8B model from about 53% to 99% on multi-step tasks.
- The post says the paper covers 97 configurations, 18 scenarios and 50 runs each.
- An ablation in the post finds retry nudges and error recovery carry the most weight.
- Zambelli reports one Mistral-Nemo 12B model scored 7% on one backend and 83% on another.
- The post proposes a ToolResolutionError, to separate returning data from finding nothing.
- Asked about overfitting, Zambelli concedes one person wrote both guardrails and evals.
- Commenter simonw notes that "guardrails" has several meanings and here means validating tool calls.
What evidence it gives
The main evidence is the author's own paper, eval harness and dashboard. The post says the harness and dashboard let readers reproduce every number. Commenters add informal reports of trying Forge with Qwen, DeepSeek and small Bonsai models. One commenter posts a table of counts from a test run.
Interests and limits
The author wrote both the guardrails and the eval suite, and says so. The post says paper numbers predate version 0.6.0. The paper, repo and dashboard were not read. Commenter results are anecdotes. The author says the evals stress-test error recovery, not end-to-end agent quality.