SIMBA (Stochastic Introspective Mini-Batch Ascent) is a DSPy optimizer for improving prompts through measured revisions rather than hand-editing instructions. It builds a pool of candidate prompts and repeatedly adds either one rule or one demonstration to a selected candidate. Which optimizer works best still depends on the task and language model. section 6.4 section 6.9
Figure 6.3 SIMBA loops over a fixed number of iterations, each time selecting one of the prompts in the pool of current candidate prompts, favoring the most promising prompts. It then adds either a rule or a demo to this and evaluates the new prompt on a minibatch. This is then added to the pool of candidate prompts along with a description of how well it performed on the minibatch.
The loop favors promising prompts, evaluates each revised prompt on a minibatch, and returns the revision and its score to the candidate pool. section 6.4
Optimize an airline intent classifier
Configure `SIMBA(metric=validate_answer, num_candidates=3, max_steps=20, max_demos=10)`, then call `simba.compile(student=intent_classifier, trainset=train_set)`. The compiled classifier can then predict the intent of a new passenger message. `num_candidates` controls prompts tested per iteration; `max_steps` controls iterations; `max_demos` caps demonstrations in a prompt. At the cap, adding a demonstration requires dropping one. section 6.4.1
SIMBA uses the program’s full source code and a history of prompts and scores to guide revisions. Its repeated reflection can be expensive: the airline classifier run took about 12 minutes, 1.5 million tokens, and 4,900 LM calls in Google Colab. The cost depends on settings and data. section 6.4.1
Supply at least one minibatch of training examples—32 by default, adjustable with `bsize`. `compile()` accepts a training set but no validation set. Scores reported during optimization may reflect overfitting, so evaluate the compiled program on a separate dataset before trusting its performance. section 6.4.1
