liveBook · Manning

Chain-of-thought prompting

As explained in Retrieval Augmented Generation, The Foundational Ideas

Chain-of-thought reasoning is a reasoning technique that generates or follows intermediate steps between a problem and its answer.

Chain-of-thought prompting inserts intermediate reasoning steps between a task and its answer. In LLM evaluation, these steps give the evaluator a procedure for inspecting claims, comparing evidence, and assigning a score rather than guessing directly. section 14.3.1

A G-Eval-style application combines three elements:

  • Define the evaluation task and criteria, such as faithfulness on a 1–5 scale. section 14.3.1
  • Append evaluation steps that identify claims, compare them with context, and count supported versus unsupported claims. section 14.3.1
  • Use rating-token probabilities to calculate a continuous weighted score, reducing the loss of uncertainty caused by a single integer rating. section 14.3.1

Faithfulness evaluation

Given source context and an answer, ask the model to identify every factual claim, check whether each claim follows from the context, count supported claims, and then assign a 1–5 score. Probability-weighted ratings can distinguish an answer that is probably a 4 from one that is confidently a 4. section 14.3.1

Listing 14.3 G-Eval-style faithfulness evaluator with chain-of-thought.
def geval_faithfulness(
    llm_client, context: str, answer: str
) -> dict[str, any]:
    """Evaluate faithfulness using G-Eval-style CoT prompting."""
    prompt = f"""You will assess the faithfulness of an answer
against its source context.
 
Evaluation Criteria:  #A
Faithfulness (1-5) - whether all claims in the answer
can be inferred from the context. A score of 5 means
every claim is fully supported. A score of 1 means
most claims are fabricated or contradicted.
 
Evaluation Steps:  #B
1. Identify each factual claim in the answer.
2. For each claim, check if the context supports it.
3. Count supported vs unsupported claims.
4. Assign a score from 1 to 5.  #C
 
Context: {context}
Answer: {answer}
 
Provide your evaluation steps, then your score (1-5)."""
    response = llm_client.generate(prompt)
    return {"evaluation": response}  #D
 

Inspect a compact G-Eval-style Python evaluator showing the prompt structure and explicit reasoning steps. section 14.3.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains prompt structure, evaluation steps, and probability-weighted scoring, but stops short of broader reasoning internals.
Examples
Demonstrates coherence scoring, faithfulness checking, explicit claim verification, and continuous 1–5 probability-weighted ratings.
What you can do after
Apply G-Eval-style prompting to construct an evaluator that produces explicit checks and more finely grained scores.
How it connects
Connects chain-of-thought prompting with RAG, faithfulness evaluation, coherence evaluation, and token probabilities.

Also explained in

Compare how 29 books explain chain-of-thought prompting →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.