Chain-of-thought prompting inserts intermediate reasoning steps between a task and its answer. In LLM evaluation, these steps give the evaluator a procedure for inspecting claims, comparing evidence, and assigning a score rather than guessing directly. section 14.3.1
A G-Eval-style application combines three elements:
- Define the evaluation task and criteria, such as faithfulness on a 1–5 scale. section 14.3.1
- Append evaluation steps that identify claims, compare them with context, and count supported versus unsupported claims. section 14.3.1
- Use rating-token probabilities to calculate a continuous weighted score, reducing the loss of uncertainty caused by a single integer rating. section 14.3.1
Faithfulness evaluation
Given source context and an answer, ask the model to identify every factual claim, check whether each claim follows from the context, count supported claims, and then assign a 1–5 score. Probability-weighted ratings can distinguish an answer that is probably a 4 from one that is confidently a 4. section 14.3.1
Listing 14.3 G-Eval-style faithfulness evaluator with chain-of-thought.
def geval_faithfulness(
llm_client, context: str, answer: str
) -> dict[str, any]:
"""Evaluate faithfulness using G-Eval-style CoT prompting."""
prompt = f"""You will assess the faithfulness of an answer
against its source context.
Evaluation Criteria: #A
Faithfulness (1-5) - whether all claims in the answer
can be inferred from the context. A score of 5 means
every claim is fully supported. A score of 1 means
most claims are fabricated or contradicted.
Evaluation Steps: #B
1. Identify each factual claim in the answer.
2. For each claim, check if the context supports it.
3. Count supported vs unsupported claims.
4. Assign a score from 1 to 5. #C
Context: {context}
Answer: {answer}
Provide your evaluation steps, then your score (1-5)."""
response = llm_client.generate(prompt)
return {"evaluation": response} #D
Inspect a compact G-Eval-style Python evaluator showing the prompt structure and explicit reasoning steps. section 14.3.1




























