liveBook · Manning

Prompt programming

As explained in Building LLM Applications with DSPy

A programming paradigm in which developers specify language-model inputs, expected outputs, and evaluation criteria in code rather than manually authoring prompts.

Prompt programming specifies a language-model task in code: its inputs, expected responses, and evaluation criteria. A framework such as DSPy generates prompts and supports their evaluation and optimization, letting developers focus on the task rather than its precise wording. section 1.0

This differs from prompt engineering, where developers manually rephrase and test prompts. Manual tuning can become difficult to track and may need to be repeated for each model. Prompt programming instead supports clean, testable, modular code and repeatable experiments. section 1.1.2 section 1.2.2

A practical workflow is to:

  • Define the task and build a baseline application. section 1.8
  • Collect representative test data and define a function that consistently evaluates responses. section 1.2.5
  • Evaluate the baseline, then generate and evaluate candidate prompts. Optimization can refine candidates over multiple iterations. section 1.8
  • Re-execute the process when testing another model or prompting technique, or when prompts need retuning. section 1.2.5 section 1.8

Optimizing an intent classifier

For a customer-service intent classifier, an optimizer can generate candidate prompts, evaluate each against a criterion, and select the best performer. In an illustrative comparison of three candidates, the second scores highest at 90%; further iterations can modify and reevaluate prompts. section 1.1.2

Figure 1.2 Prompt optimization for a customer service intent classifier. The optimization process generated three candidate prompts, each of which is evaluated. In the end, the best prompt is selected. In this case, that is the second prompt, which has the highest score: 90%. DSPy also supports processes that modify and reevaluate prompts over several iterations.
figure

The diagram shows how candidate prompts are evaluated and one is selected for the classifier. section 1.1.2

The approach is especially useful for complex or long-running applications, and automated search can make smaller models more practical. It still requires test data, code, and framework knowledge; for casual interactions where quality is less critical, a manually written prompt may suffice. section 1.2.7 section 1.4 section 1.2.5

How Building LLM Applications with DSPy covers it

How deep it goes
Outlines a baseline, evaluation, and iterative optimization workflow, including candidate generation and selection.
Examples
Illustrates candidate selection with a customer-service intent classifier and discusses inconsistent article summaries as a motivation.
What you can do after
Apply it via DSPy to evaluate and optimize prompts for language-model tasks.
How it connects
Connects prompt programming with prompt engineering, evaluation, optimization, and modular software development.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building LLM Applications with DSPy. The section links point to the parts of the book it draws on.