liveBook · Manning

Pipeline

As explained in Go by Example

A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.

A pipeline organizes processing as ordered stages: each stage receives a message, performs focused work or forwards it, and passes the result to the next stage. In a concurrent Go pipeline, stages run concurrently and communicate through channels, making data flow explicit. section 1.4.3 section 6.6.1

This structure separates a larger job into maintainable units. Stages can be added, removed, or reused without rewriting the other stages, while channels link their inputs and outputs. The final stage delivers results to a consumer. section 6.6.1 section 6.11

Build a concurrent pipeline by following the message flow:

  • Start with a producer that creates messages for the first channel. section 6.6.2
  • Insert optional transforming or regulating stages, such as a throttler that limits request flow when an RPS option is enabled. section 6.8.2
  • End with a dispatcher or other consumer-facing stage that performs work and sends results onward. section 6.6.2
  • Use a coordinator to connect stage channels and return the pipeline output. section 6.7.2

Concurrent HTTP requests

A request client can preserve a synchronous-looking SendN API while running a producer, optional throttler, and dispatcher internally. The producer emits requests, the throttler optionally slows them, and dispatcher goroutines send HTTP requests. Results flow through a channel and are pushed to consumers through a Results iterator as they arrive. section 6.6.2 section 6.1.2 section 6.8.2

Treating concurrency as an implementation detail lets an API move from sequential to concurrent execution without changing client code. This improves usability, but a pipeline must also handle consumers that stop early: otherwise goroutines can leak. Cancellation propagation addresses that reliability concern. section 6.11 section 7.1

Figure 6.3 SendN runs the pipeline and returns a Results iterator. Stages run concurrently and are connected through channels. Dispatcher goroutines call the configured Send function to send concurrent HTTP requests and return results.
A diagram of a process AI-generated content may be incorrect.

The diagram traces SendN from request production through throttling and dispatch to results returned by an iterator. section 6.6.2

How Go by Example covers it

How deep it goes
Works through designing and integrating producer, optional throttler, and dispatcher stages, while deferring basic goroutine and channel knowledge.
Examples
Uses an HTTP load-testing client whose producer, throttler, and dispatcher send requests and stream Result values.
What it assumes
Assumes basic Go concurrent programming with goroutines and channels.
What you can do after
Build it from scratch by composing channel-connected stages that concurrently process requests without changing the exported API.
How it connects
Connects channels, goroutines, iterators, synchronous APIs, fan-out, directional channels, and cancellation propagation.
Cover of Go by Example
liveBookmanning.com
Read Go by Example on liveBook Get the book at manning.com

Also explained in

Compare how 48 books explain pipeline →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Go by Example. The section links point to the parts of the book it draws on.