liveBook · Manning

Human-in-the-loop

As explained in Building Specialized AI Systems

Human-in-the-loop is a system design approach that incorporates human review or intervention into an automated data or model workflow.

Human oversight adds review to an automated labeling workflow: a language model generates synthetic summaries, then people verify and refine them before training. section 2.5.2

A practical workflow uses these steps:

  • Clean analyst reports into consistent text records. section 2.5.2
  • Prompt a language model to generate concise, professional summaries focused on findings, metrics, and conclusions. section 2.5.2
  • Review and refine generated summaries, then store the approved outputs as a training-data column. section 2.5.2
Listing 2.25 Example training prompt
SUMMARY_PROMPT_TEMPLATE = """\
Given the following analyst report, generate a concise, informative summary 
in a professional tone. 
Focus on the main findings, key metrics, and conclusions. Do not add any 
commentary beyond what is in the report.
 
Report:
{report_text}
"""

Inspect the prompt template that constrains generated summaries and reduces irrelevant commentary before human review. section 2.5.2

How Building Specialized AI Systems covers it

How deep it goes
Details automated generation and subsequent verification, but does not specify reviewer criteria or approval procedures.
Examples
Uses analyst reports, generated summaries, prompt templates, and a DataFrame summary column as concrete workflow artifacts.
What it assumes
Assumes cleaned source documents and a configured language-model API, while demonstrating label generation and review.
What you can do after
Apply it via automated summary generation followed by human verification before using outputs for training.
How it connects
Connects synthetic data generation with data labeling, prompt engineering, and fine-tuning.

Also explained in

Compare how 28 books explain human-in-the-loop →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.