13 Pipeline design

 

In this chapter

  • ensuring required actions happen regardless of failures
  • speeding up pipelines by running tasks in parallel instead of sequentially
  • reusing pipelines by parameterizing them
  • weighing the tradeoffs when deciding how many pipelines to use and what to put into each
  • identifying the features you need in a CD system to express pipeline graphs

Welcome to the last chapter of Grokking Continuous Delivery. In this chapter, I’ll show the overall structure of continuous delivery (CD) pipelines, and the features you need to look for in CD systems in order to structure your pipelines effectively.

A big piece of this story is CD systems that optimize for reuse. This approach builds on our overall theme of config as code—specifically, using software design best practices when writing CD pipelines, just like you would with any other code.

PetMatch

To help you understand the importance of pipeline design, I’ll show the pipelines used at PetMatch. This company provides a service that allows potential pet owners to search for pets available for adoption. The company’s unique matchmaking algorithm helps match the best pet to the best owner.

The company is no longer a start-up and has been operating for just over eight years. During this time, its has put a lot of emphasis on CD automation, so it has built up multiple pipelines across various services.

Matchmaking CD pipelines

CD pipeline problems

End-to-end test pipeline

End-to-end test pipeline and errors

Finally behavior

Finally as a graph

Finally in the matchmaking pipeline

End-to-end test pipeline and speed

Conclusion