liveBook · Manning

NLI

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Natural language inference is the classification of the logical relationship between a premise and a hypothesis as entailment, contradiction, or neutrality.

Natural language inference (NLI), also called Recognizing Textual Entailment, classifies the logical relationship between a premise and a hypothesis as entailment, contradiction, or neutral. It verifies whether evidence supports, refutes, or fails to determine a claim. section 5.1.1

Use NLI when a reference defines what is true, such as retrieved documents, databases, or records. In hallucination detection, treat evidence as the premise and a generated claim as the hypothesis. section 5.1.1 section 5.2.3

A practical NLI verification workflow is:

  • Split generated responses or sampled responses into individual sentences or claims. section 5.2.3
  • Compare each evidence sentence with the target claim using an NLI model. section 5.2.3
  • Count entailments and contradictions: entailments reduce hallucination risk, while contradictions increase it. section 5.2.3

Catching a numerical contradiction

A semantic-similarity method may treat “revenue increased by 15%” and “revenue increased by 51%” as similar. NLI identifies them as contradictory claims, exposing the factual error. section 5.2.3

Table 5.2 Comparison of SelfCheckGPT variants
Variant
Accuracy
Cost
Latency
Best for
BERTScore
Moderate
Low
Fast
High-volume screening
NLI
Good
Low-Medium
Medium
Detecting contradictions
QA
Good
Medium
Medium
Fine-grained claim verifications
LLM-as-a-Judge
Highest
High
Slow
High-stakes evaluation

Compare NLI with BERTScore, question-answering verification, and LLM-as-a-Judge methods by accuracy, cost, latency, and intended use. section 5.2.3

Figure 5.2 A production hallucination detection pipeline. Generated responses are decomposed into claims, routed to the appropriate verifier, aggregated into response-level metrics, and converted into actions such as logging, review, regeneration, or blocking.

Trace how claim decomposition, verifier routing, metric aggregation, and operational actions fit into a production hallucination-detection pipeline. section 5.1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines three relationship labels and applies them to sentence-level hallucination scoring through SelfCheckGPT.
Examples
Uses sampled responses as evidence and contrasts revenue increases of 15% and 51% to demonstrate contradiction detection.
What it assumes
Assumes sentence-level claims and reference evidence, while defining entailment, contradiction, and neutrality explicitly.
What you can do after
Diagnose unsupported or contradictory claims with NLI models in closed-domain applications and lightweight online checks.
How it connects
Connects semantic similarity, BERTScore, SelfCheckGPT, LLM-as-a-Judge, fact-checking, and closed-domain hallucination detection.

Also explained in

Compare how 2 books explain NLI →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.