A behavior gap occurs when a model has the relevant facts but responds in the wrong format, tone, or level of detail. For example, a correct answer may arrive as a paragraph when the application requires JSON, or its tone may vary between runs. Retrieving more documents will not by itself fix that behavior. section 4.5.1 section 4.5.9
Table 4.4 Knowledge gap signals versus behavior gap signals
|
Knowledge gap
signals
|
Behavior gap
signals
|
|
Model says “I don’t
know” or hedges on a question that has a definite answer.
|
The answer is
factually correct, but in the wrong format (paragraph instead of JSON, prose
instead of bulleted list)
|
|
Wrong factual answer
with high confidence (a hallucinated date, name, or number)
|
The answer is
correct, but the tone is inconsistent across runs (formal one time, casual
the next)
|
|
References
documentation, APIs, or product names that do not exist
|
The model follows
the output schema sometimes and ignores it other times
|
|
Answers about events
after the training cutoff or about private internal documents
|
Succeeds on simple
cases and fails on complex cases that draw on the same underlying knowledge
|
|
Same query,
different model versions, different factual answers
|
Same query,
different prompt phrasings, large variability in style or structure
|
Compare factual failures with signs that the model has the knowledge but uses it inconsistently. section 4.5.1
Distinguish behavior from missing knowledge before choosing an adaptation technique. Paste relevant source documents into the prompts for ten failing cases and rerun them. A quality jump points to a knowledge gap and supports retrieval-augmented generation (RAG). If quality stays flat, the problem may be behavioral; inspect whether the answers use the supplied facts but still fail the output requirements. section 4.4.6 section 4.5.1
Choose the least costly intervention that addresses the observed failure:
- Try few-shot prompting for classification, summarization, or extraction. Five to eight well-chosen examples often deliver much of the quality of fine-tuning, with inference-token costs. section 4.10
- For a format-only gap, use a JSON schema or constrained decoding before training; these can enforce structure without changing model weights. section 4.5.1
- Consider parameter-efficient fine-tuning for behavior that a schema cannot express, such as voice or tone, particularly at high request volumes. section 4.5.1 section 4.10
- Use RAG when the model lacks the needed facts. A fine-tuned model can still lack those facts, while a RAG pipeline can retrieve the right document and still produce the wrong format. section 4.5.1
Set a prompt-tuning budget before iterating. If three consecutive iterations fail to improve held-out accuracy beyond the validator’s noise band and run-to-run variability obscures their differences, switch approaches rather than continuing to tune the prompt. section 4.4.6
Contract review
A contract-review system can use RAG to supply current regulatory text and fine-tuning to produce structured tables and follow a firm’s risk-rating taxonomy. The two techniques address different gaps and can be used together. section 6.5
