Using an LLM as a judge

Some results have no answer key. Which of two summaries is better? Is this list of emerging topics actually useful? Traditionally, people judge, which is slow and expensive. The idea behind LLM-as-a-judge is to have a strong language model do the judging, and explain its reasoning while it’s at it. The paper that tested this properly is Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023).

Three ways to ask

• Pairwise comparison: “Here’s a question and two answers. Which is better, or is it a tie?” The most reliable, but the number of pairs grows quickly as you compare more systems.
• Single-answer grading: “Score this answer from 1 to 10.” Scales easily, but is less sensitive to small differences, and scores can drift.
• Reference-guided: grading with a correct answer to compare against. Needed for maths and reasoning (more below).

Does it agree with people?

The authors built two benchmarks: MT-Bench, 80 two-turn questions across writing, roleplay, extraction, reasoning, maths, coding, STEM and the humanities, judged by 58 experts; and Chatbot Arena, a public site where anyone chats with two anonymous models side by side and votes for the better one. Then they compared GPT-4’s verdicts with the humans’:

GPT-4 agreed with the experts more often than the experts agreed with each other. When an expert disagreed with GPT-4 and was shown its reasoning, they found it reasonable 75% of the time and changed their own vote 34% of the time. Agreement also depends on how different the answers are: from about 70% on close calls to nearly 100% when one answer is clearly better. And the cheaper single-answer grading agreed with people almost as well as pairwise comparison did.

Its biases

• Position bias: preferring whichever answer comes first. The fix is cheap: ask twice with the order swapped, and only count verdicts that agree.

• Verbosity bias: preferring longer answers, whether or not the extra length adds anything.
• Self-enhancement bias: preferring answers written by itself, possibly also by models of the same family.
• Weak at grading maths and reasoning: a judge that can solve a problem itself will still often approve a wrong answer it’s shown. Asking it to solve the problem first helped a little; grading against a reference answer (its own, worked out separately) cut the failures from 70% to 15%.
• Long conversations: judging a whole conversation in one prompt worked better than judging each turn separately.

Cheaper judges

A small open model, Vicuna-13B, was a poor judge out of the box: its verdicts changed when the answers were swapped most of the time. Fine-tuned on 20,000 human votes from Chatbot Arena, it reached 85.5% agreement with people, close to GPT-4’s 87% on the same test. A small model trained on human judgements can become a cheap, local judge.

How I’d use it

For evaluating emerging-topic detection, where there’s no answer key:

1. Check the judge first. Have people label a small sample and make sure the LLM agrees with them about as often as they agree with each other. That one step is what made this paper credible.
2. Prefer pairwise questions, like “is method A’s topic list or method B’s more useful?”, over absolute scores.
3. Swap the order and keep only consistent verdicts.
4. Don’t reward length. A longer topic description shouldn’t win for being longer.
5. Give references where you can, like a timeline of known events when replaying past news.
6. Spend the human time on close calls, where the judge is least reliable.

One caveat: these were 2023 models, and some of the weaknesses may have shrunk since. The checks are cheap enough to keep anyway.