liveBook · Manning

Classifier

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A task that assigns an input to one of a predefined set of categories.

A classifier assigns an input to one category from a predefined set. Its output is discrete, making performance measurable against labeled examples, unlike free-form generation that may require subjective judgment or parsing. section 1.0 section 1.1.1

Use classification to make a model task easier to verify:

  • Convert an open-ended generation task into a multiple-choice question with a fixed option set. section 1.1.1
  • Prompt the model to select one option, then record its choice and, when available, each option's log probability. section 1.1.1
  • Use constrained decoding or structured output interfaces when unrestricted responses would complicate extraction. section 1.1.1
  • Compare predictions with labeled answers and report accuracy rather than relying on subjective assessment of generated text. section 1.0

Multiple-choice evaluation

For “What is the capital of France?”, assign the input to A, B, C, or D. Selecting C, Paris, produces a directly verifiable classification result; its higher log probability can also indicate confidence. section 1.1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines category assignment and contrasts it with free-form generation, but does not develop classifier internals or training procedures.
Examples
Uses an MMLU-style France-capital question with four options, selected answers, and option-level log probabilities.
What you can do after
Evaluate/understand model choices through accuracy, selected categories, and optionally confidence from log probabilities.
How it connects
Connects classification with evaluation, labeled data, multiple-choice prompting, constrained decoding, and log probability.

Also explained in

Compare how 36 books explain classifier →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.