Evaluation pipeline
An evaluation pipeline is an organized sequence of steps for assessing a system or its outputs against defined criteria and producing evaluation results.
Effective Conversational AI
An evaluation pipeline is an organized sequence of data preparation, system execution, measurement, and analysis steps for assessing a model or application.
- How it is used
- Structures RAG testing around business objectives, measurable metrics, configuration comparisons, and iterative refinement.
- How deep it goes
- Explains metric selection and comparative analysis, but stops short of implementation details.
- Examples
- Uses PharmaBot configurations and business-specific chatbot scenarios to compare accuracy, latency, throughput, and resource use.
- What you can do after
- Evaluate RAG configurations, identify trade-offs, and guide continued experimentation and tuning.
Read how Effective Conversational AI explains evaluation pipeline →
Build an AI Agent (From Scratch)
An evaluation pipeline is an automated sequence that submits evaluation cases to models, validates responses, and records results and metadata.
- How it is used
- Automates repeatable comparison of multiple models across many evaluation problems.
- How deep it goes
- Details schemas, rate limits, retries, validation, exception capture, concurrency, and result grouping.
- Examples
- Uses GAIA problems, exact answer comparison, provider semaphores, and four selected models over twenty problems.
- What you can do after
- Build it from scratch as an asynchronous evaluator that produces correctness, prediction, refusal, and error records.
Read how Build an AI Agent (From Scratch) explains evaluation pipeline →
Build a Reasoning Model (From Scratch)
An ordered process that loads a model, generates outputs, and scores them against a dataset or reference answers.
- How it is used
- Quantifies model performance by generating outputs and scoring them against reference answers across many examples.
- How deep it goes
- Explains the end-to-end workflow and answer verification, but does not specify implementation code or scoring formulas.
- Examples
- Uses math problems, generated final answers, reference solutions, and calculator-like symbolic verification as concrete evaluation artifacts.
- What you can do after
- Evaluate model accuracy quantitatively and distinguish correct solutions from merely convincing responses.
Read how Build a Reasoning Model (From Scratch) explains evaluation pipeline →
Related concepts
- asyncio.gatheruses · asyncio.gather is a Python asyncio function that runs multiple awaitable objects concurrently, waits for them to complete, and collects their results.
- generationuses · Generation is the computational production of outputs from supplied inputs or context, often by applying a model to produce predictions or new content.
- experimentationused by · Experimentation is the deliberate testing of an idea, intervention, or alternative by observing outcomes to gather evidence or compare effects.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.