liveBook · Manning

Pipeline

As explained in Data Analysis with LLMs

A data-processing pipeline is an ordered sequence of computational stages that transform or analyze data, with each stage passing results to the next.

A data-processing pipeline connects steps so that one step’s result becomes input to the next. It is useful when a task needs more than a single language-model call—for example, when data must be prepared, analyzed, and turned into a usable output. section 10.1 section 10.0

Classify customer feedback

A simple text-classification pipeline takes a comment, inserts it into a prompt asking for “Positive” or “Negative,” sends the prompt to GPT-4o, and parses the reply into an answer string. The same chain can process a batch of comments. In a complete Python script, comments come from a CSV file and the results are saved in a new class column. section 10.2.1 section 10.2.2 section 10.2.3

from langchain_core.runnables.passthrough import RunnablePassthrough
chain = ({'text':RunnablePassthrough()} | prompt | llm | parser)

The Python expression connects input mapping, prompt template, model, and parser in the order each result is needed. section 10.2.2

The same pattern supports different kinds of work:

  • Speech translation records audio, transcribes it, translates the text into a requested language, then generates spoken output. section 7.4.1
  • Question answering over large data loads and divides documents, indexes their chunks, retrieves relevant items, and includes them with the question in a model prompt. section 10.6.1

Start with a few clear steps and check what each one receives and returns. LangChain calls such connected components a chain; its chains can also support parallel or conditional steps, though neither is needed for the simple classifier. The illustrated setup uses specific package versions because LangChain interfaces may change. section 10.2.1 section 10.1

How Data Analysis with LLMs covers it

How deep it goes
Works through a three-component classification chain and its CSV application, while deferring parallel and conditional execution.
Examples
Uses sentiment classification from a CSV file, speech translation, and indexed-data question answering.
What it assumes
Introduces the chain components before assembly and provides package-installation instructions for running the Python example.
What you can do after
Apply it via LangChain to classify a batch of input texts and save the labels alongside CSV data.
How it connects
Links prompt templates, language models, output parsers, batch processing, and retrieval.

Also explained in

Compare how 48 books explain pipeline →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data Analysis with LLMs. The section links point to the parts of the book it draws on.