How deeply does Building LLM Applications with DSPy cover ensemble?

Specifies program compilation, inference-time execution, and reduction, but does not detail how to build the constituent programs.

Ensemble

As explained in Building LLM Applications with DSPy

A modeling technique that combines outputs from multiple models or programs.

An ensemble runs multiple programs and reduces their outputs to one result. Combining imperfect programs can improve quality, at the cost of running them all in production. section 6.6

In DSPy, an ensemble is assembled by:

Passing a list of programs to the Ensemble optimizer; the programs may use different optimizations, language models, or modules. section 6.6
Defining a reduce function that combines their predictions. Majority vote is one option; a custom function can return a label only when all programs agree, and otherwise return “Uncertain.” section 6.6
At inference time, the resulting program executes the listed programs and applies the reducer; compilation itself takes virtually no time. section 6.6

Listing 6.15 Calling the `Ensemble` optimizer with a custom reduce function

def reduce_func(predictions):
    intent_labels = [p.intent_label for p in predictions]
    num_unique_labels = len(set(intent_labels))
    if num_unique_labels == 1:
        return dspy.Prediction(intent_label=intent_labels[0])
    return dspy.Prediction(intent_label="Uncertain")

ensemble_optimizer = Ensemble(reduce_fn=reduce_func)
prog_ensemble = ensemble_optimizer.compile([prog_copro, prog_mipro,
                                            prog_infer_rules])
print(prog_ensemble(message="What meals are served on the 3pm flight to
                    Madrid?"))

A custom reducer for classifier predictions returns the shared intent label when all programs agree, or “Uncertain” otherwise. section 6.6

Table 6.5 Tracking the predictions to create a strong ensemble

	Ex 1	Ex 2	Ex 3	Ex 4	Ex 5
Prog 1	Correct	–	–	Correct	Correct
Prog 2	Correct	Correct	Correct	–	–
Prog 3	Correct	–	Correct	Correct	Correct
Prog 4	Correct	–	–	Correct	–
Prog 5	–	Correct	Correct	Correct	–
Majority	Correct	Correct	Correct	Correct	Correct
Majority of 1, 2, 5	Correct	Correct	Correct	Correct	Correct

Prediction results across examples show how tracking program strengths can help identify a useful subset for an ensemble. section 6.6

How Building LLM Applications with DSPy covers it

Cover of Building LLM Applications with DSPy

liveBook manning.com

Read Building LLM Applications with DSPy on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Outlier Detection in PythonAn ensemble is a combination of multiple models or detectors whose outputs are combined to obtain more robust results than an individual model or detector.
liveBook manning.com

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA technique that combines multiple models or their predictions to improve predictive results.
liveBook manning.com

Ensemble Methods for Machine LearningAn ensemble is a predictive model formed by combining the outputs of multiple component models.
liveBook manning.com

Statistics Every Programmer NeedsAn ensemble model combines predictions from multiple constituent models to produce a collective prediction.
liveBook manning.com

Experimentation for EngineersA model ensemble is a collection of predictive models whose predictions or decisions are combined, compared, or otherwise used collectively.
liveBook manning.com

Deep Learning with Python, Third EditionA technique that combines multiple models or views to improve performance by exploiting complementary errors or representations.
liveBook manning.com

Machine Learning Algorithms in DepthAn ensemble method combines predictions or decisions from multiple models to obtain a joint predictive result.
liveBook manning.com

Deep Learning with R, Third EditionA model ensemble is a collection of predictive models whose outputs are combined to produce a joint prediction.
liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA collection of multiple models whose predictions are combined, typically to improve accuracy or robustness.
liveBook manning.com

Generative AI in ActionA model ensemble is an architecture that combines multiple machine-learning models to improve overall capability, performance, or reliability.

Compare how 11 books explain ensemble →

Related concepts

random forestincludes · A random forest is an ensemble model that combines predictions from multiple decision trees built with randomized data samples, feature selections, or both.
gradient boostingincludes · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
baggingincludes · Bagging, or bootstrap aggregation, is an ensemble technique that trains multiple models on bootstrap samples of data and combines their predictions.
language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
decision treeuses · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
diversityuses · Diversity is the degree of variation among members of a set or population in relevant characteristics, perspectives, behaviors, or outcomes.

This explainer was generated by Manning's knowledge graph from the text of Building LLM Applications with DSPy. The section links point to the parts of the book it draws on.