liveBook · Manning

Multi-call debate

As explained in AI Model Evaluation

A debate-style evaluation technique that uses separate model calls for the competing advocates and a final judging call.

Multi-call debate evaluates competing responses through isolated advocate calls followed by a separate judging call. Isolation prevents one model from constructing both sides and quietly favoring one argument. section 10.3.3

A typical evaluation uses three calls:

  • Call 1 sees the input and argues for Response A without seeing the opposing argument. section 10.3.3
  • Call 2 sees the input and argues for Response B without seeing the first argument. section 10.3.3
  • Call 3 sees the finished arguments, not the raw responses, and judges which case withstands scrutiny better. section 10.3.3

The technique reduces self-preference and argument-leakage risks, but increases cost and makes advocate assignment a confounding variable. Unequal advocate strength can make the evaluation measure arguing ability rather than response quality. section 10.3.3

How AI Model Evaluation covers it

How deep it goes
Defines a three-call workflow and its bias trade-offs, but stops short of prescribing assignment controls or validation procedures.
Examples
Uses competing Response A and Response B arguments passed through two advocates and one final judge.
What you can do after
Evaluate competing responses with separated advocate and judge calls while recognizing cost and assignment-bias risks.
How it connects
Connects LLM-as-a-judge with prompting, self-preference bias, verbosity bias, and evaluation confounding.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.