liveBook · Manning

Fidelity

As explained in LLM Customization and Fine-Tuning

A measure of whether a sentiment label matches the content of its sentence.

Fidelity measures whether a sentiment label matches its sentence. In a financial training-data quality gate, a judge scores fidelity from 1 to 5; a score below 4 fails the gate. section 3.6.4

Use fidelity as one part of a three-dimension check:

  • Require fidelity, format, and domain safety scores of at least 4; a strong fidelity score alone does not pass an example. section 3.6.4
  • Treat the gate as a filter for candidate examples, not as a measure of the teacher model’s overall quality. section 3.6.4
Listing 3.5 Specifying the judging system prompt
JUDGE_SYSTEM_PROMPT = """You are a quality reviewer for a financial
language model training dataset.
 
Score this financial sentiment example on three dimensions (1 to 5 each):
 
FIDELITY (1 to 5): Does the sentiment label match the sentence content?
  5 = label is clearly correct for this sentence
  1 = label contradicts the sentence
 
FORMAT (1 to 5): Is this realistic financial text?
  5 = reads like real financial news or analyst commentary
  1 = sounds artificial, generic, or not financial
 
DOMAIN SAFETY (1 to 5): Are financial facts plausible?
  5 = all percentages, rates, amounts are realistic
  1 = contains impossible or implausible financial figures
 
A PASS requires ALL THREE >= 4.
 
Return ONLY valid JSON, no preamble:
{"fidelity": X, "format": X, "domain_safety": X, "pass": true_or_false}"""

The judging prompt defines the 1–5 fidelity scale and requires all three quality scores to reach 4 for a pass. section 3.6.4

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines a 1–5 label-match score and a pass threshold of 4 within a three-part gate, but does not assess teacher-model quality.
Examples
Scores whether a financial sentence’s sentiment label fits, alongside checks for realistic financial writing and plausible facts.
What it assumes
Assumes a candidate sentence and sentiment label to compare, and evaluates them alongside format and financial-fact plausibility.
What you can do after
Apply it via an LLM-as-judge rubric to accept or reject candidate sentence-label examples.
How it connects
Connects sentiment labels with dataset quality control, format checks, and domain-safety checks.

Also explained in

Compare how 11 books explain fidelity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.