liveBook · Manning

Reliability

As explained in Rearchitecting LLMs

Reliability is the consistency with which a model produces correct or constraint-compliant outputs across inputs.

Reliability here means producing medical-extraction responses that satisfy a defined JSON schema. A field-by-field validator distinguishes parseable JSON from responses that meet the full contract and records failure reasons. section 7.3.2

Establish a baseline before changing model weights: test schema compliance and general capabilities. With a strict prompt, the baseline reached 85% compliance on 40 samples; irrelevant inputs were a notable weakness at 58.3%. Treat these results as directional, not statistically robust. section 7.3.2

                     samples  valid_json_pct  schema_ok_pct
category                                             
abbreviations        6            100.0             100.0
clean                8            100.0             100.0
implicit             7            100.0             100.0
irrelevant           12            58.3              58.3
typos                7            100.0             100.0

The category breakdown shows where baseline schema compliance failed. section 7.3.2

After specialization, QLoRA and QDoRA both reached 100% compliance with a minimal instruction, including on irrelevant inputs. This improved reliability while reducing the prompt needed, but came with a moderate decline in general benchmarks, most pronounced on ARC-Easy. section 7.3.5

How Rearchitecting LLMs covers it

How deep it goes
Defines field-level JSON checks and compares 40-sample baseline results with post-training compliance, while treating small-sample results as directional.
Examples
Tests medical-information extraction across clean, abbreviated, implicit, irrelevant, and typo-containing notes.
What you can do after
Evaluate schema adherence with field-level JSON validation and compare specialized models against a prompted baseline.
How it connects
Connects schema compliance to prompt engineering, QLoRA, QDoRA, and general-capability benchmarks.

Also explained in

Compare how 21 books explain reliability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.