liveBook · Manning

Accuracy

As explained in Vibe Engineering

Correctness is the degree to which software behavior satisfies its intended specification and real-world requirements.

A fluent answer or a successful type check can still be wrong. Accuracy measures how often output is correct against a standard external to the model. For classification, it is the proportion of predictions that match the expected labels. Neither a model’s confidence nor structurally valid code proves correctness. section 1.3 section 8.2.1 section 4.6.3

Overall accuracy can hide the errors that matter most. A medicine-risk classifier reports over 94% accuracy, but precision and recall reveal different risks: false alarms and missed high-risk cases. Check those measures and the confusion matrix before treating the overall score as evidence of safety. section 4.6.3

Figure 4.10 The classification accuracy: confusion matrix shows that most (931) of the actual false values were predicted as false, and most of the true values (491) were predicted as true.

The confusion matrix separates correct safe and high-risk predictions, making the aggregate accuracy easier to interpret. section 4.6.3

Context is not a free accuracy upgrade

In a text-to-SQL benchmark, adding recent queries improved overlap similarity, but gains diminished. For moderate queries, the median rose from about 0.39 with no history to 0.56 with ten queries and 0.61 with twenty. Overlap similarity is the reported proxy here, not proof that generated SQL is correct. For this benchmark, ten queries offered the preferred accuracy-to-cost trade-off; twenty cost more for a smaller gain. section 6.4.1 section 6.4.2

More context can also make an answer less reliable. In one code-comprehension benchmark, a model scored 29% at 32,000 tokens and 3% at 256,000, even though the answer was present in both contexts. An advertised context window is therefore not a guarantee that all supplied information will be used reliably. section 3.4.1

Use accuracy as one part of a verification loop:

  • Define an external standard and test representative cases; do not rely on fluent output or compilation alone. section 1.3 section 8.2.1
  • Inspect error types and task-specific measures alongside the aggregate score, especially when false negatives and false positives have different costs. section 4.6.3
  • Benchmark context choices against both quality and cost. Treat the best setting as specific to the workload rather than a universal maximum. section 6.4.1 section 6.4.2

How Vibe Engineering covers it

How deep it goes
Defines accuracy and works through context-size benchmarks, while distinguishing type-checking signals from proof of correctness.
Examples
Compares a medicine-risk classifier’s predictions and text-to-SQL overlap scores at different query-history lengths.
What you can do after
Evaluate and understand model accuracy using task-specific error measures and quality-versus-cost benchmarks.
How it connects
Connects accuracy with precision, recall, confusion matrices, context windows, latency, and cost.

Also explained in

Compare how 49 books explain accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.