Evaluator

As explained in AI Agents and Applications

An evaluator is a model or component that assesses outputs against specified criteria and produces judgments, scores, comparisons, or feedback.

How AI Agents and Applications covers it

How deep it goes

Names three evaluation criteria and dataset-based testing, but does not describe evaluator internals.

Examples

Uses relevance, correctness, and sensitivity as criteria for evaluating LLM completions.

What you can do after

Evaluate LLM completions with built-in checks for relevance, correctness, and sensitivity.

How it connects

Connects evaluators to LLM completions, datasets, continuous testing, and regression testing.

liveBookmanning.com

Read AI Agents and Applications on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  AI Agents in Action, Second EditionLLM-as-judge is an evaluation technique in which a language model assesses another model's output against specified criteria.
• 

  liveBookmanning.com
  Building Reliable AI SystemsAn evaluation technique in which a large language model assesses the quality of another model's output.
• 

  liveBookmanning.com
  The Art of AI Product DevelopmentLLM-as-a-Judge is an evaluation technique in which a large language model assesses generated outputs against specified criteria.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringLLM-as-judge is an evaluation technique in which one large language model assesses the outputs of another model.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackLLM-as-a-judge is an evaluation technique in which a language model assesses the quality or preference of model outputs according to provided criteria.
• 

  liveBookmanning.com
  Build an AI Agent (From Scratch)LLM-as-a-judge is an evaluation method in which a language model scores or classifies another system's outputs according to an evaluation rubric.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionAn LLM judge is a language model configured to assess or classify the outputs of another model according to specified criteria.
• 

  liveBookmanning.com
  Building LLM Applications with DSPyA language model assigned to assess the quality of responses produced by another language model.

Compare how 9 books explain evaluator →

Related concepts

• language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• evaluation datasetuses · An evaluation dataset is a collection of examples or test cases, often with expected or judged outcomes, designated to assess a model or system against defined criteria.
• coherencemeasures · Coherence is the degree to which the parts of a text, reasoning process, or system fit together consistently and form a connected, intelligible whole.
• metric functionused by · A metric function is a function that evaluates a language-model prediction against an example and optionally a call trace to produce an assessment score.
• accuracymeasures · Accuracy is the degree to which an output, prediction, or measurement agrees with a correct or reference outcome under a specified criterion.

This explainer was generated by Manning's knowledge graph from the text of AI Agents and Applications. The section links point to the parts of the book it draws on.