liveBook · Manning

Prompt sensitivity

As explained in Machines that Think

Prompt sensitivity is the tendency of a language model's outputs or performance to change in response to superficial prompt variations.

Prompt sensitivity is the tendency for model outputs or measured performance to change when wording or formatting changes, even if the task’s meaning stays the same. section 6.3.2 section 6.3.3

Surface variations that can affect results include:

  • Label formatting, separators, example order, and layout. section 6.3.2
  • Tone and politeness, even when questions and answer options remain unchanged. section 6.3.2
  • Equivalent prompt formulations whose effects differ across models. section 6.3.3

Tone can shift measured accuracy

In one multiple-choice study using ChatGPT-4o, accuracy rose from 80.8% for very polite prompts to 84.8% for very rude prompts. This result illustrates a measured difference, not a general rule that rudeness improves performance. section 6.3.2

Sensitivity varies with the model and the measurement method; some apparent effects may reflect rigid scoring. Prompt revisions without objective feedback can seem beneficial while leaving accuracy unchanged or reducing it. section 6.3.3 section 6.3.3

How Machines that Think covers it

How deep it goes
Explains controlled prompt variations and evaluation findings, but does not provide a dependable optimization procedure.
Examples
Compares formatting, example order, layout, politeness, and semantically equivalent prompts.
What you can do after
Evaluate prompt changes cautiously by separating preserved task meaning from changes in measured performance.
How it connects
Links prompt sensitivity to prompt design, model evaluation, classification accuracy, and scoring methods.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.