liveBook · Manning

Advocate-strength effect

As explained in AI Model Evaluation

An apparent evaluation difference caused by one assigned advocate arguing more effectively than another rather than by a quality difference between the evaluated responses.

The advocate-strength effect occurs when a debate-style judge favors the response defended by the stronger advocate, making advocacy skill look like response quality. A single debate cannot separate these effects from position bias. section 10.3.3

Test whether a verdict survives changes to the evaluation setup:

  • Swap advocate roles so each model argues for each response. section 10.3.3
  • Swap the order in which the judge sees the arguments. section 10.3.3
  • Aggregate the runs, report agreement frequency, and send low-agreement cases to human review. section 10.3.3

Assignment reversal

If Response A wins when model_X advocates for it but Response B wins after the assignments reverse, the result indicates advocate strength rather than a reliable quality difference. section 10.3.3

How AI Model Evaluation covers it

How deep it goes
Tests role and order reversals, aggregates repeated verdicts, and routes low-agreement cases to human review.
Examples
Uses reversed advocate assignments and reversed argument order to test verdict stability.
What it assumes
Assumes debate-style judging with multiple advocate models, a judge, and two candidate responses.
What you can do after
Diagnose unstable debate verdicts and reserve repeated panels for launch gates, safety boundaries, or unresolved disagreements.
How it connects
Connects to position bias, panel evaluation, verdict aggregation, and human review.

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.