Plausible training examples can still contain incorrect labels or invented facts. A quality gate checks candidates against explicit requirements before they enter a dataset, filtering a larger pool rather than measuring the generator’s overall quality. section 3.6.4
For synthetic data, generate variations anchored to real seed examples, then gate them before mixing them with real data. Judge errors against the target task: an invented hostname may be harmless in a classification example but dangerous in a summary that asserts it as fact. Production logs can also supply candidates for a gate. section 3.4.3 section 3.4
A financial sentiment gate scores each candidate from 1 to 5 on three dimensions:
- Fidelity: Does the sentiment label match the sentence? section 3.6.4
- Format: Does the sentence resemble financial news or analyst commentary? section 3.6.4
- Domain safety: Are percentages, rates, and amounts plausible? section 3.6.4
Rejecting a plausible but unsafe example
A candidate scoring 5 for fidelity, 5 for format, and 3 for domain safety fails: all three scores must be at least 4. Rejecting 15–25% of candidates is described as normal; scoring the whole pool and discarding the bottom tier is preferred to relying on a perfect generation prompt. section 3.6.4
Listing 3.6 Scoring function for one example
def score_one_example(example: dict) -> dict:
content = (
f"Sentence: \"{example['sentence']}\"\n"
f"Label: {example['label']}"
)
response = client.messages.create(
model=TEACHER_MODEL,
max_tokens=100,
system=JUDGE_SYSTEM_PROMPT,
messages=[{"role": "user", "content": content}],
)
try:
scores = json.loads(response.content[0].text.strip())
except json.JSONDecodeError:
# If the judge fails to produce valid JSON, treat as failed
scores = {"fidelity": 0, "format": 0, "domain_safety": 0, "pass": False}
return {**example, "scores": scores, "verified": scores.get("pass", False)}The scoring function submits the sentence and label to a judge, records its returned pass flag, and treats invalid JSON as a failure. section 3.6.4
A candidate gate is distinct from a deployment validation gate. After retraining, a model must also meet evaluation and safety thresholds, avoid category-level regressions greater than 3% relative, and produce expected canary outputs; a failed check blocks deployment. section 9.7.3








