liveBook · Manning

Schema compliance

As explained in Rearchitecting LLMs

Schema compliance accuracy is the proportion of model outputs that conform to a required structured output schema.

Schema compliance is the proportion of model outputs that conform to a required structured-output contract. It measures whether responses have the required structure, not whether extracted facts are correct. section 7.3.2

For clinical extraction, the contract specifies JSON fields for patient age, symptoms, vital signs, medications, and duration. Validation distinguishes parseable JSON from a response that also matches the schema; a failure reason identifies what went wrong. A parseable response alone is not enough to count as compliant. section 7.3.2

Listing 7.6 Schema validation
REQUIRED_SCHEMA = {
    "patient_age":  (int, type(None)),
    "symptoms":     list,
    "vital_signs":  dict,
    "medications":  list,
    "duration_days": (int, type(None)),
}
 
VITAL_SIGNS_KEYS = {"temperature", "heart_rate", "blood_pressure"}
 
def check_schema_compliance(response_text: str) -> dict:
    cleaned = response_text.strip()\
                      .removeprefix("```json")\
                      .removesuffix("```").strip() #A
 
    try:
        parsed = json.loads(cleaned)
    except json.JSONDecodeError as e:
        return {
            "is_valid_json":  False,
            "matches_schema": False,
            "failure_reason": f"JSON parse error: {e}",
        }
 
    for key, expected_type in REQUIRED_SCHEMA.items(): #B
        if key not in parsed:
            return {
                "is_valid_json":  True,
                "matches_schema": False,
                "failure_reason": f"Missing key: '{key}'",
            }
        if not isinstance(parsed[key], expected_type):
            return {
                "is_valid_json":  True,
                "matches_schema": False,
                "failure_reason": f"Wrong type for '{key}': "
                                  f"got {type(parsed[key]).__name__}",
            }
 
    for vk in VITAL_SIGNS_KEYS:  #C
        if vk not in parsed["vital_signs"]:
            return {
                "is_valid_json":  True,
                "matches_schema": False,
                "failure_reason": f"Missing vital_signs key: '{vk}'",
            }
 
    return {
        "is_valid_json":  True,
        "matches_schema": True,
        "failure_reason": None,
    }
 

Inspect the validator’s required fields, expected types, JSON parsing, and failure reporting. section 7.3.2

Compare a strict-prompt baseline with fine-tuning

On 40 clinical notes, a 4-bit SmolLM2-1.7B-Instruct model achieved 85% compliance with a fifteen-line prompt. The category breakdown reports 58.3% compliance on 12 irrelevant-input samples and 100% in the other categories. After QLoRA training, the model reached 100% on the same test set with the prompt “Extract:”. Both QLoRA and QDoRA reached 100% in the reported comparison. These small-sample results are directional, not statistically robust. section 7.3.2 section 7.3.3 section 7.3.5

Compare compliance alongside general-capability benchmarks: specialization improved the measured format reliability while producing a moderate benchmark drop, most pronounced on ARC-Easy. To investigate data efficiency, train on progressively smaller subsets while keeping the test set intact, then inspect which categories fail first. section 7.3.5 section 7.5

How Rearchitecting LLMs covers it

How deep it goes
Implements JSON and schema checks with failure reporting, then evaluates aggregate and category-level results.
Examples
Uses clinical-note extraction into JSON and compares clean, abbreviated, implicit, irrelevant, and typo-containing inputs.
What you can do after
Evaluate structured-output reliability with a validator and compare model variants across input categories.
How it connects
Connects schema validation with prompt engineering, QLoRA, QDoRA, and general-capability benchmarks.

Also explained in

Compare how 2 books explain schema compliance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.