Chapter 6: Building a software-defined engineering platform

Introduction

This introduction explains why Epetech needs an engineering platform before the chapter begins building one.

• Epetech’s platform motivation: Explain how Epetech’s lack of a clear platform strategy creates delivery and operational problems for its developers and customers.
  Why it matters: These problems explain why Epetech has tasked a team with creating a better engineering platform.

Key terms: Epetech, tech silos, lead-time planning, product incidents

6.1 Building our own example engineering platform

The section explains why an example engineering platform starts as a simple new product, prioritizes user experience, and organizes its domains to support later growth.

• Simple working platform foundation: Justify starting an engineering platform as a simple, working new product rather than designing a complex system all at once.
  Why it matters: It explains why the example begins with an MVP rather than attempting a complete platform from the outset.
• Existing technology adoption friction: Explain why an organization’s existing engineering technologies may not add up to an effective developer platform.
  Why it matters: Having the technologies already in place does not mean developers can adopt and use them easily.
• Evolving platform domain ownership: Describe how ownership of the platform product’s internal domains can change as it grows.
  Why it matters: Clear, loosely coupled boundaries let the product begin with one team without requiring that team to own every domain forever.

Key terms: Gall’s law, brand-new product, MVP, user experience, API access, self-serve experiences, traditional IT silos, product ownership, internal domain boundaries, subdomains

6.2 Prerequisites to getting started

This section identifies the tools and cloud resources needed to start an example engineering platform and explains the bootstrap and cost constraints.

• Bootstrap tools and challenge: Explain which tool capabilities are needed initially and why starting from scratch creates a bootstrap challenge. Describe how SaaS tools help address it.
  Why it matters: These prerequisites make it possible to begin delivering the platform using the same software-defined approach intended for later work.
• Cloud resource cost planning: Explain why the AWS learning exercises require cost planning and how a learner can limit costs or proceed without cloud-provider access.
  Why it matters: Planning for resource access and spending makes the exercises practical to undertake.
• Example tool selection: State which tools the reference examples use for the initial requirements and distinguish those examples from mandatory tool choices.
  Why it matters: The reader can follow the samples closely without mistaking their specific vendors for requirements of platform engineering.

Key terms: source code version control, secrets store, infrastructure state store, pipeline orchestrator, bootstrap challenge, SaaS tools, AWS free tier, de-provisioning resources, Minikube, GitHub

6.2.1 Getting started with the example tools

This section sets up the example platform’s development tools and explains how to avoid making automation depend on one person’s access token.

• Example tool access setup: Describe how to prepare GitHub, Terraform Cloud, CircleCI, and a secrets vault for the Epetech exercises, including the access credentials needed for later automation.
  Why it matters: These accounts and credentials provide the starting access for the example platform exercises.
• Automation credential continuity: Explain why automation tied to a personal access token can break when its owner changes roles or leaves, and distinguish the service-account and OIDC-token alternatives described here.
  Why it matters: Choosing credentials that do not depend on one employee helps keep pipelines running through team changes.

Key terms: GitHub organization, platform-team, Terraform Cloud, CircleCI, 1Password vault, access token, personal access token, service accounts, machine users, team secrets store

6.2.2 Developer tools selection criteria

This section explains how to select developer tools that support a secure, maintainable self-service platform, with specific criteria for secrets managers and pipeline tools.

• Platform-oriented tool selection: Explain why developer preferences should inform tool selection but cannot be the only criterion, and why secure SaaS tools can be preferable to self-managed ones.
  Why it matters: A familiar tool can still be a poor platform choice if it makes self-service difficult or creates an ongoing maintenance burden.
• Secrets access and onboarding: Describe how a secrets manager should fit the platform’s customer identity model and automated team onboarding.
  Why it matters: Identity-aligned onboarding lets teams use secrets without the platform team manually arranging access for each person or integration.
• Pipeline-ready secrets management: Describe the secrets-manager capabilities needed for both automated pipelines and developers managing their own secrets.
  Why it matters: These capabilities let developers update secrets while automated workflows consume them consistently and securely.
• Repository-owned pipeline definitions: Explain why teams should own software-defined CI and deployment pipelines in their repositories, with shared code adopted by version.
  Why it matters: Teams can change their delivery workflows without waiting on another team or receiving an unexpected breaking update.
• Event-driven pipeline changes: Explain why the section favors code-defined pipeline changes and event-based triggers over manual pipeline creation and routine manual triggering.
  Why it matters: Predictable, code-defined changes and triggers make automated delivery easier to trust.
• Prepared and accessible runners: Describe why a pipeline tool should support prepared runners, debugging access, and persistence of data between jobs.
  Why it matters: Prepared, debuggable runners save developer time across repeated pipeline runs.

Key terms: self-serve experience, SaaS development tools, platform engineering considerations, customer identity model, authentication and authorization, team onboarding, role-based access permissions, service account, OIDC trust, CLI

6.3 Infrastructure pipeline orchestration practices

Introduces pipeline-managed software definitions for platform components and notes that their paths to production vary by component.

• Pipeline-managed platform components: Explain that every engineering-platform component is software defined, most of those definitions are managed through pipelines from initial testing to production, and the environments along that path depend on the part of the platform being configured.
  Why it matters: It sets the context for why the chapter discusses different kinds of infrastructure pipelines.

Key terms: software defined, pipeline managed, path to production

6.3.1 Account-level pipelines

Defines account-level pipelines by the once-per-account work they perform and the environment they target.

• Account-level pipeline scope: Explain that account-level pipelines automate actions or infrastructure configurations once per account, with each account treated as an environment.
  Why it matters: This scope distinguishes account-level work from other platform pipeline work.

Key terms: account-level pipelines, once-per-account, environment

6.3.2 Control plane-level pipelines

This section explains how control plane pipelines move Kubernetes-based platform changes through staged environments before production.

• Staged control plane promotion: Explain why the platform maintains several control plane instances along its path to production.
  Why it matters: Separate stages let platform engineers validate changes before developers encounter them in production.
• Cluster environment roles: Describe what counts as an environment for Kubernetes cluster-level features, including when production uses multiple clusters.
  Why it matters: Thinking in terms of cluster roles keeps the path to production meaningful as the platform grows.
• Cluster-wide application promotion: State which cluster changes follow the control plane path to production.
  Why it matters: The promotion path applies to both the clusters and the platform-managed applications running across them.

Key terms: control plane-level pipelines, path to production, promotion process, Kubernetes, clusters, production, environment, provisioning, configuration, cluster-wide applications

6.3.3 Namespace-level pipelines

The section explains how preview namespaces protect developers’ release paths and applies software-development practices to infrastructure pipelines built for the platform product.

• Preview environment purpose: Explain why the platform provides developers with preview namespaces outside their usual path to production and what a failure there reveals.
  Why it matters: Preview exposes the effects of platform changes while developers can continue their normal releases.
• Responses to preview failures: Describe how a preview failure can guide the response to an upcoming platform change, including a change that developers themselves must accommodate.
  Why it matters: Finding a breaking change early gives both platform builders and developers room to respond.
• Stable infrastructure release path: Justify a fixed, recurring infrastructure release path through test environments, and distinguish the platform team’s first test environment from a customer environment.
  Why it matters: A dependable infrastructure release process helps keep the environments developers rely on fit for continuous deployment.
• Software practices for infrastructure: Describe how the section applies test-driven development, frequent trunk integration, and repository quality checks to infrastructure code.
  Why it matters: Treating infrastructure as software brings proven checks and development habits to platform changes.
• Maintainable pipeline boundaries: Explain how configuration parameters, external scripts, and a secrets manager keep infrastructure pipelines reusable and easier to change.
  Why it matters: Separating environment configuration, pipeline ordering, reusable logic, and secrets avoids duplication and reduces tool lock-in and credential-handling risks.
• Platform-focused infrastructure architecture: Justify designing the infrastructure codebase for the engineering platform rather than every unrelated organizational use case.
  Why it matters: A clear product boundary helps the platform team deliver useful capabilities without recreating a slow traditional IT process.

Key terms: preview role, preview instance, namespace, path to production, preview environment, feature flag, breaking change, fixed, recurring release path to production, environmental stability, DEV

6.3.4 Choosing an IaC framework

The section explains how to choose infrastructure-as-code tools for cloud infrastructure and why a platform product should keep its Terraform code focused on its own use case.

• Cloud IaC framework fit: Explain why declarative IaC suits most cloud infrastructure work and what qualities the section recommends in a framework, while recognizing when an imperative tool is appropriate.
  Why it matters: The choice of IaC framework should fit how cloud services actually accept and apply configuration.
• Product-focused Terraform design: Distinguish Terraform code for an opinionated platform product from code meant to serve varied team requests, and explain what that difference means when designing code or assessing outside modules.
  Why it matters: Keeping Terraform aligned with the platform product makes it easier to change without taking on unrelated use cases.

Key terms: declarative, imperative, IaC, cloud vendor APIs, processing overhead, Terraform, internal product, opinionated use case, Terraform modules, resource definitions

6.3.5 Test-driven development of infrastructure code

This section explains how test-driven development and integration tests help verify infrastructure configuration, acceptance criteria, reusable modules, and configuration drift.

• Infrastructure test-first workflow: Describe the test-driven development workflow for infrastructure code and explain why writing tests before implementation helps keep them aligned with the intended outcome.
  Why it matters: A passing test is useful only if it checks the outcome you actually wanted.
• Reusable module integration tests: Explain why integration tests are valuable for a reusable infrastructure module even when a cloud provider performs the provisioning.
  Why it matters: The module’s choices need verification even if the provider API behaves correctly.
• Acceptance criteria verification: Describe how a configuration test can demonstrate that an infrastructure story’s acceptance criteria were met.
  Why it matters: Successful provisioning alone does not show that a story’s specific requirements were satisfied.
• Recurring configuration drift detection: Explain how recurring integration tests can detect infrastructure configuration drift and why drift is possible.
  Why it matters: Infrastructure can change after deployment without a corresponding change to its code.
• Infrastructure testing tool examples: Identify the configuration-testing tools illustrated in the section and describe the basic role-check exercise.
  Why it matters: The examples make the idea of a configuration test concrete, including what a deliberate failure looks like.

Key terms: test-driven development (TDD), tests written after primary development, false positives, reusable module, Terraform module, integration tests, CI pipeline, story acceptance criteria, tag values, naming convention

6.3.6 Static code analysis

The section explains which static checks are useful for Terraform IaC and how to run them locally, in commit hooks, and in CI.

• Useful IaC static checks: Explain why static analysis of declarative IaC emphasizes syntax, style, best practices, and security rather than traditional imperative-code measures.
  Why it matters: It helps you choose checks that provide useful feedback for infrastructure code.
• Terraform analysis tool roles: Describe what Terraform CLI validation and formatting, Tflint, and Trivy each check, and how to use their feedback to correct a Terraform file.
  Why it matters: Separating the tools’ roles makes their feedback easier to act on.
• Local commit scan feedback: Explain why the section recommends running static scans through local git commit hooks as well as in CI, and describe the kinds of checks in a typical Terraform commit scan.
  Why it matters: Commit-time checks can catch problems before a CI pipeline fails.

Key terms: IaC, static code analysis, test coverage, cyclomatic complexity, Terraform CLI, validation, fmt, Tflint, Trivy, trivy config

6.3.7 Reusable pipeline code

The section explains how to keep infrastructure pipelines simple by moving reusable behavior into shared, versioned code while preserving room for local customization.

• Dumb pipelines and smart code: Explain what belongs in a repository’s local pipeline and what belongs in shared code. Describe how source changes normally trigger pipelines and how local jobs determine the order of work.
  Why it matters: This division keeps each repository’s pipeline understandable without duplicating implementation details.
• Shared pipeline library value: Justify maintaining shared pipeline libraries as a platform product team, including how they support common requirements and reduce duplicated pipeline work.
  Why it matters: Shared libraries let a platform team support many teams’ pipelines without taking ownership of every local pipeline.
• Shared job customization points: Describe how injection points and parameter or tool-version overrides keep shared jobs useful without requiring a library change for every team-specific need.
  Why it matters: Flexible, guarded shared jobs reduce the chance that developers abandon them or wait on a maintainer for routine changes.
• Terraform customization examples: Describe how the shared Terraform plan job examples use inserted steps and optional arguments to accommodate runtime values and unusual Terraform needs.
  Why it matters: The Terraform example shows what flexible shared jobs look like when a pipeline needs values computed at runtime.
• Lifecycle oriented pipeline resources: Distinguish broad shared workflows from smaller tool-specific components and describe the role of maintained runners.
  Why it matters: Teams benefit both from ready-to-use lifecycle workflows and from smaller supported pieces when they need a custom pipeline.

Key terms: dumb pipelines and smart code, local pipelines, triggers, steps, shared pipeline code libraries, orbs, DRY, nonfunctional requirements, versioning, injection points

6.3.8 Private executors (runners)

This section explains why organizations using SaaS pipeline tools may need private runners, the resource tradeoff they introduce, and a local CircleCI experiment.

• Private runner networking tradeoff: Explain why an organization might use private runners with a SaaS pipeline tool even though the book recommends securing public-facing access through zero-trust networking. Describe the efficiency problem that comes with running executors on the organization’s own cloud resources.
  Why it matters: The choice of runner must account for both private-network requirements and efficient pipeline execution.
• Local private runner experiment: Describe the proposed CircleCI experiment: start a local Kubernetes cluster and use CircleCI’s container-agent service to create a private runner that watches configured projects and runs their pipelines in containers in that cluster.
  Why it matters: The local setup makes the relationship between a configured CircleCI project and a privately run pipeline concrete.

Key terms: private executors, private runners, zero-trust networking, self-managed resource, CircleCI, container-agent service, private runner resource, Kubernetes

6.4 Cloud administrative identity

This section defines the cloud administrative identity domain and places the engineering platform within a cloud provider’s account structure.

• Cloud administrative identity scope: Explain that the cloud administrative identity domain covers the top-level cloud provider organization and account structure, plus the identities and permissions of service accounts used for foundational infrastructure automation.
  Why it matters: It sets the boundary for what this platform product domain is responsible for.
• Platform domain account structure: Describe the platform’s own top-level domain account, its role in holding product-wide configurations, and the subaccounts for test, nonproduction, and production contexts. State that vendors use different names for these account units.
  Why it matters: It shows where the platform sits in the cloud organization before the individual environments are considered.

Key terms: cloud administrative identity domain, service accounts, foundational infrastructure automation, domain account, product-wide configurations, subaccounts, accounts, projects, subscriptions

6.4.1 Top-level Domain Account

Explains why an engineering platform may need a top-level cloud account for product-wide settings and what access its delivery team needs.

• Top-level account purpose: Explain how a top-level domain or state account provides one place for platform-wide governance policies and configurations that affect multiple cloud accounts.
  Why it matters: It helps keep platform-wide rules and shared settings consistent as the platform grows.
• Platform team administrative access: Explain why the engineering platform delivery team needs administrator-level permissions for the cloud accounts where its product is created, even when other teams manage some top-level items.
  Why it matters: Clarifying administrative access lets the platform team manage its accounts even when ownership of top-level items is shared.

Key terms: top-level domain or state account, product-wide configuration, governance policies, delivery team, administrator-level permissions, cloud accounts

6.4.2 Platform product test environments

Distinguishes platform engineers’ test environments from customer environments and explains why their accounts must be isolated.

• Platform test account isolation: Explain who uses platform product development and test environments, why their accounts are separate from customer-facing accounts, and why the account is often called a sandbox rather than a Dev account.
  Why it matters: Separate accounts let platform engineers test changes without disrupting the people who use the platform.

Key terms: platform product test environments, sandbox account, internal customer-facing accounts, account-level settings

6.4.3 Internal customer nonproduction and production environments

This section explains why a platform separates developer nonproduction environments from production while treating every internal customer-facing environment as a production service, and notes that its basic account structure may need to evolve.

• Customer-facing environment production expectations: Explain why the platform separates nonproduction developer environments from production, and why platform engineers treat all internal customer-facing environments as Production even when customers use them for development.
  Why it matters: A development label describes the customer's use of an environment, not a lower standard of platform support.
• Account structure as starting point: Describe why the basic account structure is a starting point rather than a fixed design.
  Why it matters: The initial account structure need not accommodate every future constraint from the outset.

Key terms: nonproduction developer environments, production environments, internal customer-facing environments, Production, basic structure of accounts, scale, regulatory or contractual problems

6.4.4 Service accounts and permissions

This section builds the IAM identities, roles, and pipelines used to manage platform infrastructure, rotate credentials, and release changes safely.

• Service Account Role Boundaries: Explain how infrastructure pipeline service accounts obtain permissions and why each pipeline has a dedicated role.
  Why it matters: This separation gives each pipeline the access it needs without giving its identity broad, permanent permissions.
• Initial Pipeline Bootstrap: Trace why the first roles-and-service-accounts pipeline must be bootstrapped manually and what must exist before it can take over.
  Why it matters: Bootstrapping resolves the dependency between a pipeline and the credentials it needs to run.
• Shared Pipeline Code: Explain how the example pipeline uses versioned shared jobs and commands while allowing pipeline-specific setup.
  Why it matters: Versioned shared code makes common pipeline improvements reusable without making every upgrade a waiting period for developers.
• Safe Credential Rotation: Explain why service account credential rotation uses two keys and how newly launched jobs obtain the current credential.
  Why it matters: Rotation should limit credential age without interrupting pipelines already in progress.
• Controlled Production Release: Describe how IAM pipeline changes move to production and why environmental differences require a code change.
  Why it matters: Plan review and a common code path make production changes deliberate and consistent across accounts.
• Scheduled Integration Checks: Explain why the section adds a scheduled pipeline run and how its trigger differs from an ordinary code change.
  Why it matters: Scheduled checks can reveal drift even when no one has pushed a code change.

Key terms: IAM role, service account, assume roles, dedicated roles, bootstrap, IAM users, service account credentials, secrets store, Terraform orb, versioned pipeline code

6.5 Summary