A pipeline divides work into ordered stages: each stage transforms an input item and passes its result to the next. Stages can process different items at the same time, even when every individual item must follow the same dependency order. This raises throughput for work such as document processing, audio filtering, and image or video processing. section 10.2.4 section 10.4
In Go, a stage is commonly a goroutine that receives from an input channel and returns an output channel. This input-to-output shape makes stages easy to connect: URL generation feeds page downloading, downloading feeds word extraction, and extraction can feed later frequency counting. Closing an output channel signals that a stage has no more values to send; a quit channel lets stages stop early. section 9.2.2 section 9.2.2
A web-page processing pipeline
Start with generateUrls, which emits RFC document URLs. Pass its channel to downloadPages, then pass downloaded page text to extractWords. The composition extractWords(quit, downloadPages(quit, generateUrls(quit))) expresses the flow directly: each function accepts its upstream channel and returns its downstream channel. section 9.2.2 section 9.2.2
A reusable pipeline node can receive a value of type X, apply a mapping function, and emit a value of type Y. Wiring nodes around a cupcake workflow—prepare tray, add mixture, bake, add toppings, and box—allows separate cupcake batches to occupy separate stages concurrently, unlike completing one batch sequentially before beginning another. section 10.2.4 section 10.2.4
Evaluate a pipeline with two different performance measures:
- Throughput is the number of completed items per unit of time. The slowest stage dictates it, so improve the bottleneck first. section 10.2.5
- Latency is the time for one item to travel through all stages. It is the sum of stage times, so speeding any stage can reduce it. section 10.2.5
- Choose the target deliberately: batch processing may favor throughput, while real-time work may favor latency. section 10.2.5
This diagram captures the core stage contract: accept X, map it with a function, and emit Y for the next stage. section 10.2.4
