liveBook · Manning

Tradeoff

As explained in Vibe Engineering

A trade-off is a compromise in which improving one objective or property requires accepting a reduction in another.

A tradeoff is a compromise between objectives: improving one property can require accepting less of another. An optimization may make code faster while making it harder to maintain; without traffic measurements, that complexity may buy no meaningful improvement. section 7.1

Make the competing objectives explicit before choosing a shortcut. For an LLM-based SQL generator, adding example queries to the prompt can improve accuracy, but longer prompts cost more. The useful context size depends on the measured improvement, its marginal cost, and the service’s expected traffic. section 6.4.1 section 6.4.2

Choosing prompt context by measurement

In one benchmark, moderate-query p50 accuracy rose from about 0.39 with no examples to 0.56 with 10 and 0.61 with 20. The 316-query benchmark cost 7 units at 10 examples and 10 units at 20. Ten offered the best accuracy-to-cost ratio in that run; five cost one unit less but required accepting lower accuracy. These are benchmark-specific choices, not universal thresholds. section 6.4.1 section 6.4.2

Figure 6.4 Comparison of the p50 vs the cost of running the model.

Compare measured p50 accuracy with benchmark cost as context size changes. section 6.4.2

A tradeoff is only as reliable as its evaluation. For few-shot SQL testing, draw examples from a separate frozen pool and exclude the current query’s gold SQL. Otherwise, providing the test answer in the prompt can artificially inflate results. section 6.4

Other engineering choices expose different costs:

  • Broad AI-directed data analysis may reveal unexpected insights but also produce irrelevant analysis. Specific questions focus the output, while limiting it to a subset of possible findings. section 4.3.1
  • Performance optimization can add code, maintenance, or system complexity without improving the main workflow. Start with maintainable code when traffic and latency requirements are unknown, then measure the hot path before changing it. section 7.1 section 7.2.2
  • More metrics and broader AI-proposed changes can increase analysis, review, deployment, and monitoring effort. Run benchmarks and provide measured numbers before asking for production changes. section 7.5

How Vibe Engineering covers it

How deep it goes
Works through accuracy and cost benchmarks and gives procedural guardrails for evaluation and optimization.
Examples
Compares SQL prompt histories, drug-data analysis prompts, and optimization of a word service.
What you can do after
Evaluate tradeoffs by comparing measured accuracy, projected cost, and the complexity of proposed changes.
How it connects
Connects tradeoffs to few-shot prompting, data contamination, performance optimization, and benchmarking.

Also explained in

Compare how 11 books explain tradeoff →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.