- Author
- Madhulatha Mandarapu and Sandeep Kunkunuru, VaidhyaMegha Private Limited, India
- Published
- May 2026
- Kind
- arXiv preprint with benchmark experiments
- Original
- arxiv.org/abs/2605.26874
- Subject
- Harnesses and agent loops
What it is
Two authors from an Indian company wrote the preprint. It builds on the AssetOpsBench benchmark of maintenance questions. It asks how much the data model behind the tools matters.
What it says
- The paper says GPT-4 agents over document stores reach 65% on 139 maintenance scenarios.
- LLM-written Cypher queries over the graph lift the same model from 65% to 82-83%.
- Hand-coded handlers with no LLM score 99% on graph-answerable scenarios.
- Of 139 handler answers, 86 came from graph queries and 53 did not.
- For equipment missing from the graph, an LLM adds tagged facts, answering 81.8% of 88.
- Correcting the schema prompt alone raised one scenario type to 93%, the paper says.
- The authors say the 99% is a ceiling because they wrote the answers.
What evidence it gives
The paper gives tables of pass rates on a 139-scenario snapshot and a 467-scenario release. It adds 40 new scenarios the authors wrote and a test on 88 failure-mode scenarios. Code and result files are said to be on GitHub.
Interests and limits
The authors' email domain and code repositories belong to samyama.ai. That site makes the graph database tested. Results come from a single run at temperature 0. The comparison changes more than storage, and the judge differs from the published score. The 467-scenario result tests only the hand-coded handlers. Real sites have messier data, and this is a preprint.