Chapter 3: Classifying user intent
The introduction explains text classification and frames intent classification as mapping varied customer requests to a predefined set of intents.
After this chapter you will be able to
- User requests to intents. Explain how intent classification lets a chatbot interpret differently worded customer requests by mapping them to a limited, predefined set of intents or permitted operations.
- Baseline purpose and performance. Explain why a baseline is a relatively simple but working first application, and how its performance guides later improvement: it may be adequate for a simpler task, while poor performance signals that more effort is needed.
- Utterances and text length. Explain that intent classification usually works on short user utterances that ideally express one intent, and why classifying paragraph-length text is harder.
- Binary and multiclass classification. Distinguish binary from multiclass classification by the number of possible classes, and explain why the intent-classification problem is multiclass.
- Single-label and multilabel classification. Distinguish single-label from multilabel classification by how many labels an item receives, and describe the section's initial assumption about utterances.
- Static and dynamic classes. Distinguish static from dynamic classification by whether the set of classes is fixed or can evolve.
- ATIS Test Data Preparation. Describe where the ATIS data comes from, why this zero-shot baseline uses the test set without training or validation data, and how the downloaded data is made available for inspection.
- Descriptive Zero Shot Labels. Explain why the original short intent names are replaced with more descriptive names, especially for a zero-shot classifier.
- Single Intent Data Filtering. Describe how combined-intent labels are excluded and what happens to their rows when the new label mapping is applied.
- Minimal classifier components. Describe the language model, signature, and module specified for the example classifier, and explain that executing the module returns a Prediction without requiring a separate Prediction definition.
- Reusable prompt template. Explain that DSPy generates a reusable prompt template from the signature and module, then fills its input-field slots with the message and possible labels for a particular call. Describe how the actual prompt sent to the LM can be viewed.
- Classifier signature fields and types. Describe the intent classifier’s class-based signature: it inherits from dspy.Signature, takes a message and a list of labels as inputs, and returns an intent label. Explain how field types specify the expected inputs and output.
- Class signature prompt descriptions. Explain how class and string signatures are nearly equivalent, and why a class-based signature can produce a different prompt through its docstring and field descriptions. Describe the demonstrated effect of an output-field description requesting uppercase.
- Predict classifier program. Explain how passing the intent-classification signature to Predict creates a DSPy program, and describe the inputs and output when the program is called.
- Limits of DSPy abstraction. Distinguish the LM API calls DSPy generally handles from calls that may still require a provider’s API. Explain why testing many LMs does not mean learning every provider’s API.
- Direct API classification workflow. Trace the direct OpenAI API example: create a client, build a system prompt containing the intent labels, supply the utterance as a user message, and call create() with the model and messages to receive a response.
- Handcrafted prompt maintenance costs. Explain why DSPy signatures and modules are presented as simpler and cleaner than assembling or storing handcrafted prompts, especially as prompt variations and model-specific handling grow.
- System and user message roles. Explain how separate system and user messages clarify an LM task, and describe how DSPy populates those messages.
- Security and prompt hierarchy. Explain why user messages present a security concern and describe the relative authority of user, application system, and LM-provider prompts.
- LM provider flexibility. Explain that DSPy makes it easier to switch between LM providers, while a provider’s API is limited to that provider’s models, and why switching matters.
- Prompt construction with DSPy. Describe how a DSPy signature and module replace manually assembling the system and user messages expected by the LM.
- Advantages at application scale. Explain why direct OpenAI use remains manageable for this simple classifier but DSPy’s time and quality advantages grow in applications with many LM requests and repeated retuning.
- Static and dynamic intent labels. Distinguish a predetermined set of intents, as in ATIS, from production intents that change over time. Explain that a restaurant can accommodate menu changes by updating the possible intents in its prompt, while some new intents may be unanticipated.
- Limits of an other class. Explain why LM-based classification has an advantage with changing labels and why adding an “other” class to a traditional classifier is difficult to make reliable.
- None option for unmatched intents. Explain how to instruct the DSPy classifier to return None when a message does not match any provided choice well.
- Fixed intent label vocabulary. Explain why providing a list of labels and asking the LM to choose one is not enough to guarantee a valid label, and describe how typing the output field with Literal restricts it to specified strings.
- Multilabel versus compound intents. Distinguish assigning one label to each utterance from allowing multiple intents, and explain why compound labels are usually less practical than a multilabel approach.
- DSPy multilabel adaptation. Describe the change the section proposes for handling multiple intents in DSPy.
- Readable and detailed call history. Distinguish the readable prompt-and-response display from the more detailed LM-call history, and explain what the history’s size tells you.
- Prediction usage and cached calls. Explain what Prediction-level usage tracking reports and why a repeated cached request can report no usage.
What it builds on
- Classification prompt example (chapter 2) · The earlier classification prompt example.
- The main concepts in DSPy (chapter 2) · It assumes the basic DSPy concepts introduced earlier.
- Signatures (chapter 2) · It assumes familiarity with signatures and their fields.
- Using LiteLLM to access LMs (chapter 2) · Introduces accessing LMs through an intermediary API.
- Modules (chapter 2) · The notion of a DSPy module.
- Switching between LMs (chapter 2) · The idea of switching between LMs.
- Predictions (chapter 2) · DSPy Predictions.
- The language model (chapter 2) · LM objects make the calls whose history is recorded.
Section by section
Introduction
The introduction explains text classification and frames intent classification as mapping varied customer requests to a predefined set of intents.
- Text classification and labels: Explain that text classification assigns a piece of text a label from a set of possible classes, as illustrated by positive and negative sentiment.
Why it matters: Intent classification is easier to understand as a particular use of text classification. - User requests to intents: Explain how intent classification lets a chatbot interpret differently worded customer requests by mapping them to a limited, predefined set of intents or permitted operations.
Why it matters: This mapping turns varied customer language into requests a chatbot can handle.
Key terms: classification, label, class, sentiment classification, intent classification, intent classifier, predefined set of intents
3.1 Creating a baseline classifier
Defines a baseline as a simple working first attempt that supports iteration and helps gauge task difficulty.
- Baseline purpose and performance: Explain why a baseline is a relatively simple but working first application, and how its performance guides later improvement: it may be adequate for a simpler task, while poor performance signals that more effort is needed.
Why it matters: The baseline gives you something working to improve and an early signal of how much improvement may be needed.
Key terms: baseline, successive iterations
3.1.1 Utterances
Defines the short user messages classified for intent and explains why longer text is harder to classify.
- Utterances and text length: Explain that intent classification usually works on short user utterances that ideally express one intent, and why classifying paragraph-length text is harder.
Why it matters: It clarifies what kind of input the chapter’s airline intent examples classify.
Key terms: intent classification, user utterances, intent
3.1.2 Multiclass, multilabel, and dynamic classification
This section distinguishes three ways to describe an intent-classification problem: by the number of possible classes, the number of labels per utterance, and whether the classes can change.
- Binary and multiclass classification: Distinguish binary from multiclass classification by the number of possible classes, and explain why the intent-classification problem is multiclass.
Why it matters: The number of possible intents determines whether this is a binary or multiclass task. - Single-label and multilabel classification: Distinguish single-label from multilabel classification by how many labels an item receives, and describe the section's initial assumption about utterances.
Why it matters: A request containing two intents may need two labels even when the initial classifier assumes one. - Static and dynamic classes: Distinguish static from dynamic classification by whether the set of classes is fixed or can evolve.
Why it matters: This distinction identifies whether the available intent categories stay fixed or may change.
Key terms: binary classification, multiclass classification, classes, single-label, multilabel, utterance, intent, static classification, dynamic classification
3.2 Downloading and preparing the ATIS dataset
The section prepares ATIS airline requests for a baseline intent classifier by selecting the test data, inspecting it, and cleaning up its intent labels.
- ATIS Test Data Preparation: Describe where the ATIS data comes from, why this zero-shot baseline uses the test set without training or validation data, and how the downloaded data is made available for inspection.
Why it matters: This establishes which data the baseline classifier will use and why the other splits are not needed yet. - Dataframe Intent Inspection: Describe how the dataframe is inspected before changing its labels, including checking its size, columns, intent frequencies, and sample records.
Why it matters: Inspecting the records and intent frequencies helps reveal what needs cleaning before classification. - Descriptive Zero Shot Labels: Explain why the original short intent names are replaced with more descriptive names, especially for a zero-shot classifier.
Why it matters: For this baseline, clearer labels carry task information that examples would otherwise help provide. - Single Intent Data Filtering: Describe how combined-intent labels are excluded and what happens to their rows when the new label mapping is applied.
Why it matters: Removing combined intents keeps the baseline's classification task focused on one intent per request.
Key terms: ATIS dataset, Hugging Face, datasets Python library, zero-shot prompting, test data, pandas dataframe, user intents, value_counts(), intent names, Flight Booking Request
3.3 Building a DSPy intent classifier
This section shows how a minimal DSPy intent classifier turns a message and possible labels into a prediction using a reusable prompt template.
- Minimal classifier components: Describe the language model, signature, and module specified for the example classifier, and explain that executing the module returns a Prediction without requiring a separate Prediction definition.
Why it matters: These are the pieces needed to understand what the classifier code supplies and what DSPy does for it. - Reusable prompt template: Explain that DSPy generates a reusable prompt template from the signature and module, then fills its input-field slots with the message and possible labels for a particular call. Describe how the actual prompt sent to the LM can be viewed.
Why it matters: It explains why the same classifier program can handle different messages without writing a new prompt each time. - Classifier reuse across domains: Explain why the example is not limited to airline intents: the caller supplies the text and the possible labels, so the same code can serve another classification problem that has a list of labels.
Why it matters: It shows the practical value of keeping the classification task separate from a particular message or label set.
Key terms: language model, DSPy signature, DSPy module, Prediction, prompt template, input fields, inspect_history(), utterance, labels, labels parameter
3.3.1 The signature
This section explains how a class-based DSPy signature defines the intent classifier’s inputs and output, and how types and descriptions shape the LM’s prompt.
- Classifier signature fields and types: Describe the intent classifier’s class-based signature: it inherits from dspy.Signature, takes a message and a list of labels as inputs, and returns an intent label. Explain how field types specify the expected inputs and output.
Why it matters: The signature makes the classifier’s input–output contract explicit to DSPy and the LM. - Class signature prompt descriptions: Explain how class and string signatures are nearly equivalent, and why a class-based signature can produce a different prompt through its docstring and field descriptions. Describe the demonstrated effect of an output-field description requesting uppercase.
Why it matters: A class signature can carry instructions that change the prompt and influence the classifier’s response.
Key terms: dspy.Signature, InputFields, OutputFields, message, labels, intent_label, class-based signature, string signature, docstring, field description
3.3.2 The module
The section turns an intent-classification signature into a working DSPy baseline and explains how to try it on test messages while accounting for LM calls.
- Predict classifier program: Explain how passing the intent-classification signature to Predict creates a DSPy program, and describe the inputs and output when the program is called.
Why it matters: This is the working baseline classifier used before attempting improvements. - Test predictions and call costs: Describe how the baseline can be tried on a hardcoded message or the collected test data, and explain the LM-call implication of predicting across the full dataframe.
Why it matters: Testing many messages can multiply LM calls even when each individual call is inexpensive.
Key terms: Predict module, signature, program, message, labels, intent_label, test data, dataframe, prediction, LM call
3.4 Using the OpenAI API directly
This section explains why direct LM APIs are worth understanding and where DSPy’s abstraction does—and does not—replace them.
- Limits of DSPy abstraction: Distinguish the LM API calls DSPy generally handles from calls that may still require a provider’s API. Explain why testing many LMs does not mean learning every provider’s API.
Why it matters: This distinction helps you decide when DSPy is enough and when a provider-specific API might be necessary. - OpenAI API relevance: Explain why understanding OpenAI’s models and API can be useful even when an application uses another provider.
Why it matters: OpenAI’s influence makes its API a useful reference point beyond OpenAI models.
Key terms: DSPy APIs, API signatures, prompts, responses, embeddings, OpenAI, API, parameters, Claude, Gemini
3.4.1 Classification using the OpenAI API
The section walks through intent classification with the OpenAI API and explains why hand-built prompts become harder to maintain than DSPy signatures and modules.
- Direct API classification workflow: Trace the direct OpenAI API example: create a client, build a system prompt containing the intent labels, supply the utterance as a user message, and call create() with the model and messages to receive a response.
Why it matters: This workflow makes the work hidden by the DSPy classifier concrete. - Handcrafted prompt maintenance costs: Explain why DSPy signatures and modules are presented as simpler and cleaner than assembling or storing handcrafted prompts, especially as prompt variations and model-specific handling grow.
Why it matters: Prompt organization affects whether you can understand and safely change a classifier as it grows.
Key terms: OpenAI client, system prompt, intent labels, user message, utterance, create(), handcrafted prompt, signatures, modules, substrings
3.4.2 System and user roles
This section explains how system and user messages separate a task’s definition from its input, and why that distinction matters for security.
- System and user message roles: Explain how separate system and user messages clarify an LM task, and describe how DSPy populates those messages.
Why it matters: The distinction helps explain what instructions the LM receives and where the current input appears. - Security and prompt hierarchy: Explain why user messages present a security concern and describe the relative authority of user, application system, and LM-provider prompts.
Why it matters: Message roles help the LM handle conflicts between application instructions and potentially unsafe user input.
Key terms: system prompt, user message, Signature, Module, prompt injections, hierarchy of roles, user prompt, LM provider
3.4.3 Comparing DSPy with direct use of LM APIs
This section compares a DSPy intent classifier with direct LM API use and explains why DSPy’s conveniences matter more as applications grow.
- LM provider flexibility: Explain that DSPy makes it easier to switch between LM providers, while a provider’s API is limited to that provider’s models, and why switching matters.
Why it matters: Provider flexibility makes it easier to keep an application effective as available models change. - Prompt construction with DSPy: Describe how a DSPy signature and module replace manually assembling the system and user messages expected by the LM.
Why it matters: This removes some of the manual prompt-writing work required with a direct API call. - Structured classification output: Distinguish DSPy’s Prediction object from processing raw LM JSON results, while noting the similar Structured Outputs option offered by OpenAI.
Why it matters: Structured results reduce output-handling work, though this benefit is not entirely unique to DSPy. - Advantages at application scale: Explain why direct OpenAI use remains manageable for this simple classifier but DSPy’s time and quality advantages grow in applications with many LM requests and repeated retuning.
Why it matters: The practical case for DSPy is strongest when prompt work must be maintained across a larger application.
Key terms: LM providers, OpenAI API, signature, module, system and user messages, Prediction object, JSON, Structured Outputs, manual tuning, retune
3.5 Dynamic labels vs. static labels
This section contrasts fixed intent labels with changing production labels and explains why LMs are useful when new intents arise.
- Static and dynamic intent labels: Distinguish a predetermined set of intents, as in ATIS, from production intents that change over time. Explain that a restaurant can accommodate menu changes by updating the possible intents in its prompt, while some new intents may be unanticipated.
Why it matters: A classifier built for a fixed dataset may not reflect the intents users bring to a changing service. - Limits of an other class: Explain why LM-based classification has an advantage with changing labels and why adding an “other” class to a traditional classifier is difficult to make reliable.
Why it matters: An “other” label may sound like an easy fix, but its diverse examples make it difficult for a traditional classifier to learn.
Key terms: static set of classes, dynamic labels, intents, DSPy interface, dynamic sets of labels, LM-based classification, none of the above, other, unknown, clustering assumption
3.5.1 Dynamic intent classification
The section shows how a dynamic intent classifier can handle messages that do not fit any provided label and what can be done with those messages.
- None option for unmatched intents: Explain how to instruct the DSPy classifier to return None when a message does not match any provided choice well.
Why it matters: A None result lets the classifier signal that the available choices do not fit. - Follow-up for unmatched messages: Describe the suggested ways to handle messages classified as None in production, and the alternative of letting the LM estimate an intent label.
Why it matters: An unmatched message can be reviewed later or given an estimated label rather than silently forced into an existing choice.
Key terms: signature, docstring, None, choices, intent label
3.5.2 Static intent classification
This section explains why a listed set of intent labels does not by itself constrain an LM’s output and how a typed output field can restrict the allowed labels.
- Fixed intent label vocabulary: Explain why providing a list of labels and asking the LM to choose one is not enough to guarantee a valid label, and describe how typing the output field with Literal restricts it to specified strings.
Why it matters: A fixed label vocabulary helps keep classifier outputs within the categories the application expects.
Key terms: static intent classification, Literal, output field, fixed vocabulary
3.6 Handling multiple intents
The section explains why multilabel intent classification is preferable to compound labels, how to adapt a DSPy classifier for it, and why a successful example still needs evaluation.
- Multilabel versus compound intents: Distinguish assigning one label to each utterance from allowing multiple intents, and explain why compound labels are usually less practical than a multilabel approach.
Why it matters: Choosing how to represent intents affects whether the classifier can handle requests that combine several needs. - DSPy multilabel adaptation: Describe the change the section proposes for handling multiple intents in DSPy.
Why it matters: The classifier can be adapted to request multiple intents without introducing compound labels for every combination. - Single example limitations: Explain why the working example does not establish that multilabel classification is reliable.
Why it matters: A promising demonstration should not be mistaken for evidence of reliable performance.
Key terms: single-label, multilabel, compound intents, DSPy, docstring, output field, single test, proper evaluation, optimize the prompt
3.7 Viewing the history
This section explains how to inspect DSPy’s LM-call history, retrieve generated prompts, and track usage and costs.
- Readable and detailed call history: Distinguish the readable prompt-and-response display from the more detailed LM-call history, and explain what the history’s size tells you.
Why it matters: The readable view helps with debugging, while detailed history helps you check call volume and cost. - Generated prompts outside DSPy: Explain why retrieving prompt text from history can help when another tool makes the LM calls, and how the DSPy program produces that text.
Why it matters: History lets you see or reuse the prompt actually generated, rather than assuming the program holds a fixed template. - Separate LM object histories: Describe how histories are separated by LM object, how to start fresh for an experiment, and the default scope of the cross-object history view.
Why it matters: Keeping histories separate makes call tracking across models or experiments less confusing. - Prediction usage and cached calls: Explain what Prediction-level usage tracking reports and why a repeated cached request can report no usage.
Why it matters: An empty usage result for a cached prediction does not mean the program failed to produce an answer.
Key terms: inspect_history(), history, input and output tokens, cost, prompt text, prompt template, Adapter, LM object, lm.history.clear(), n
3.8 Summary
Where this is used later
- Evaluating the DSPy baseline model (chapter 4) · Evaluating the DSPy baseline gives a concrete measure of how well the first application performs.
- Splitting the ATIS data (chapter 4) · The later ATIS split prepares data for evaluation.
- (chapter intro) (chapter 4) · Provides the DSPy intent classifier used as the baseline.
- Creating a dataset for evaluation (chapter 4) · The airline intent classifier supplies the classification task used in this example.
- Using the Example class (chapter 4) · The intent classifier takes a message and possible labels and predicts an intent.
- Defining an evaluation metric (chapter 4) · Introduces the intent classifier’s predicted class.
- Executing the LabeledFewShot optimizer (chapter 5) · The existing DSPy intent-classifier module is the program being compiled.
- Evaluating other LMs and modules (chapter 4) · Comparing other LMs draws on the reason to try models beyond the initial choice.
Concepts explained in this chapter
- single-label categorical classificationA single-label problem is a classification setup in which each input is assigned at most one class label.
- evaluation datasetInputs and associated evaluation information used to assess a prompt or language-model system.
- Hugging Face DatasetsHugging Face Datasets is a Python library for downloading, processing, sharing, and loading machine-learning datasets.
- zero-shot promptingZero-shot prompting is a prompting technique that asks a language model to perform a task without providing task-specific demonstrations.
- signatureA declarative specification of the inputs and outputs of a DSPy language-model operation.
- promptTextual instructions or input supplied to a language model to elicit a response.
- classificationThe task of assigning an input to one or more predefined categories.
- DSPy programAn executable DSPy composition of one or more modules.
- PredictPredict is a DSPy module that represents a basic prompting operation for invoking a language model.
- moduleA DSPy module composed within another module and invoked as part of its computation.
- training datasetThe portion of a dataset used to fit or configure a machine-learning model or application.
- demonstrationAn input paired with an appropriate output to illustrate a task to a language model.
- validation setA validation set is a dataset partition used to tune choices or configurations during model development.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.