liveBook · Manning

Smoke test

As explained in LLM Customization and Fine-Tuning

A smoke test is a basic test that checks whether essential parts of a software system run successfully.

A smoke test is a small, low-cost check that catches obvious breakage before a larger evaluation or run. It can reveal whether a prompt or training workflow works at all, but it cannot establish that the system performs well. section 4.4.2 section 6.1.2

Use the check to catch failures early, not to make fine-grained quality judgments:

  • For prompt iteration, try 20 representative production inputs. This can expose a broken prompt, but the set is too small to rank close variants. section 4.4.2
  • Before full supervised fine-tuning, run two or five training steps to surface out-of-memory errors, shape mismatches, and data or configuration problems. Inspect sample generations too: a run can finish while producing empty or garbled text. section 6.1.2 section 6.8.4
  • Run code on a CPU and laptop before committing to a GPU run. section 1.9.3

Before a full training run

Set --max_steps 2, confirm the run completes, and inspect its sample output before launching the longer job. section 6.1.2

How LLM Customization and Fine-Tuning covers it

How deep it goes
Checks execution and obvious failures, but does not rank close prompt variants or replace larger release evaluations.
Examples
Uses 20 production-log inputs for prompts, short training runs for SFT, and CPU execution on a laptop.
What you can do after
Apply it with a small held-out prompt set or --max_steps 2–5 to catch failures before full evaluation or training.
How it connects
Connects prompt iteration, held-out test sets, and supervised fine-tuning.

Also explained in

Compare how 5 books explain smoke test →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.