Schema compliance is the proportion of model outputs that conform to a required structured-output contract. It measures whether responses have the required structure, not whether extracted facts are correct. section 7.3.2
For clinical extraction, the contract specifies JSON fields for patient age, symptoms, vital signs, medications, and duration. Validation distinguishes parseable JSON from a response that also matches the schema; a failure reason identifies what went wrong. A parseable response alone is not enough to count as compliant. section 7.3.2
Listing 7.6 Schema validation
REQUIRED_SCHEMA = {
"patient_age": (int, type(None)),
"symptoms": list,
"vital_signs": dict,
"medications": list,
"duration_days": (int, type(None)),
}
VITAL_SIGNS_KEYS = {"temperature", "heart_rate", "blood_pressure"}
def check_schema_compliance(response_text: str) -> dict:
cleaned = response_text.strip()\
.removeprefix("```json")\
.removesuffix("```").strip() #A
try:
parsed = json.loads(cleaned)
except json.JSONDecodeError as e:
return {
"is_valid_json": False,
"matches_schema": False,
"failure_reason": f"JSON parse error: {e}",
}
for key, expected_type in REQUIRED_SCHEMA.items(): #B
if key not in parsed:
return {
"is_valid_json": True,
"matches_schema": False,
"failure_reason": f"Missing key: '{key}'",
}
if not isinstance(parsed[key], expected_type):
return {
"is_valid_json": True,
"matches_schema": False,
"failure_reason": f"Wrong type for '{key}': "
f"got {type(parsed[key]).__name__}",
}
for vk in VITAL_SIGNS_KEYS: #C
if vk not in parsed["vital_signs"]:
return {
"is_valid_json": True,
"matches_schema": False,
"failure_reason": f"Missing vital_signs key: '{vk}'",
}
return {
"is_valid_json": True,
"matches_schema": True,
"failure_reason": None,
}
Inspect the validator’s required fields, expected types, JSON parsing, and failure reporting. section 7.3.2
Compare a strict-prompt baseline with fine-tuning
On 40 clinical notes, a 4-bit SmolLM2-1.7B-Instruct model achieved 85% compliance with a fifteen-line prompt. The category breakdown reports 58.3% compliance on 12 irrelevant-input samples and 100% in the other categories. After QLoRA training, the model reached 100% on the same test set with the prompt “Extract:”. Both QLoRA and QDoRA reached 100% in the reported comparison. These small-sample results are directional, not statistically robust. section 7.3.2 section 7.3.3 section 7.3.5
Compare compliance alongside general-capability benchmarks: specialization improved the measured format reliability while producing a moderate benchmark drop, most pronounced on ARC-Easy. To investigate data efficiency, train on progressively smaller subsets while keeping the test set intact, then inspect which categories fail first. section 7.3.5 section 7.5

