- Author
- Aviya Maimon and four co-authors
- Published
- 11 June 2026 (version 2)
- Kind
- Academic preprint with a benchmark analysis and released code
- Original
- arxiv.org/abs/2507.20208
- Subject
- Language models and training
- Themes
- Checking the work
What it is
Maimon and colleagues apply psychometric factor analysis to LLM benchmark scores. They ask whether scores reflect many abilities or a few shared ones. They also release code and a leaderboard.
What it says
- The paper applies factor analysis to a matrix of 60 models on 44 benchmarks.
- The authors find eight latent factors and label them as skills.
- Tasks that look alike can fall on different skills, such as GPQA and TriviaQA.
- Models with near-equal Chatbot Arena ratings can have very different skill profiles.
- The average skill score correlates with Arena Elo at 0.73 over 13 shared models.
- Twelve tasks were enough to estimate new models' profiles, mean error 0.095 in tests.
- The authors report PCA agrees less with Arena than their method (0.28 against 0.73).
What evidence it gives
The paper reports robustness tests on its own 60 by 44 score matrix. These include other factor counts, 30% of models held out, and leave-one-task-out runs. It checks internal consistency, with alpha 0.836. It correlates skill scores with Chatbot Arena for 13 models. It adds a second analysis on the MTEB embedding leaderboard.
Interests and limits
The matrix covers 60 instruction-tuned models, and generation tasks use an LLM judge, DeepSeek-R1. The authors say eight skills is not a unique correct resolution, and names are interpretive. A simple average of task scores correlated 0.86 with Arena, above 0.73. Newer models were reconstructed with higher error than older ones. The main text and appendices C to K were read. The references and appendices A and B were not.