- Author
- Harrison Lee, Samrat Phatale, Hassan Mansoor and others, Google DeepMind and Google
- Published
- 3 September 2024 (version 3)
- Kind
- Industry research paper with human evaluation
- Original
- arxiv.org/abs/2309.00267
- Subject
- Language models and training
- Themes
- Checking the work
What it is
Researchers at Google ask whether an LLM can label preferences instead of people. They train policies with both kinds of labels. They compare the results on summarization, helpful dialogue and harmless dialogue.
What it says
- Humans preferred RLAIF over the supervised baseline 71% of the time on summaries.
- Head to head, RLAIF and RLHF tie on summaries and helpful dialogue. On harmless dialogue RLAIF scores 88% against 76% for RLHF.
- Same-size RLAIF, with a labeler as small as the policy, still won 68%.
- Direct RLAIF skips the reward model. The LLM scores outputs from 1 to 10.
- Annotators preferred direct RLAIF over same-size RLAIF 60% of the time. The authors only hypothesize why.
- Chain-of-thought prompting generally improved labeler alignment. Extra few-shot examples lowered it for summaries and helpfulness.
- The authors estimate AI labeling costs over 10x less than human labeling.
What evidence it gives
The paper reports human evaluation on three tasks using PaLM 2 models. About 2k rating instances were each scored by three raters. After length control, RLAIF and RLHF win 59% and 61% against the baseline. A small blind check of 70 summaries found no significant difference.
Interests and limits
The authors work at Google and use Google's PaLM 2 models. Tests cover three text tasks and a small policy model, PaLM 2 Extra-Small. Combining human and AI feedback gave no gain in the authors' test. The paper says its two-policy sensitivity study cannot support firm conclusions. The authors urge extreme caution in medicine, law and employment.