liveBook · Manning

Condorcet's jury theorem

As explained in Machine Learning System Design

Condorcet's jury theorem states that independent voters who are individually more likely than not to be correct become increasingly likely to reach a correct majority decision as their number grows.

When several labelers independently judge the same binary-classification example, a majority can be more accurate than any individual labeler. If each is correct more than half the time, group accuracy generally rises as participation grows. section 6.2.4

Three labelers at 70% accuracy

Three labelers who are each correct 70% of the time produce a correct majority label in 78.4% of cases. A cost-conscious workflow can first use two labelers, accept agreement, and add three more votes only when they disagree. section 6.2.4

Use the result as a practical labeling baseline, while checking its assumptions:

  • Treat independence cautiously: labelers may share correlated errors, especially on noisy or hard-to-read samples. section 6.2.4
  • Measure consistency and mutual agreement before investing in more elaborate labeling designs. section 6.2.4

How Machine Learning System Design covers it

How deep it goes
Explains the probability intuition and a three-labeler calculation, but stops short of formal derivation.
Examples
Uses three 70%-accurate labelers and an adaptive two-then-five-vote labeling workflow.
What it assumes
Assumes basic probability and binary classification, while introducing independence and correlated-error limitations through examples.
What you can do after
Apply it via majority voting to improve labels while escalating only ambiguous cases for additional votes.
How it connects
Connects majority voting with data labeling, ensemble methods, crowdsourcing, and correlated errors.
Cover of Machine Learning System Design
liveBookmanning.com
Read Machine Learning System Design on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.