Chapter 9: Architecture changes to support scale

Introduction

The introduction frames platform scaling as a business-driven architectural and organizational decision, not simply an increase in capacity.

• Organizational and technical scaling: Explain why scaling an engineering platform requires changes to its product team, roles, processes, and architecture—not just more servers or better performance.
  Why it matters: Adding capacity alone will not keep a growing platform effective or affordable.
• Autonomous platform domain teams: Describe why a platform serving dozens or hundreds of development teams may need multiple domain teams, and what those teams need from one another.
  Why it matters: Platform teams need room to work independently, just as the developers using the platform do.
• Business-driven scaling priorities: Explain how business priorities determine when and where to scale different platform domains, using an example from the introduction.
  Why it matters: Choosing what to scale first helps the platform serve the business need rather than following a uniform expansion plan.

Key terms: engineering platform product team, platform control plane, services and extensions, domain boundaries, autonomous, self-serve, business priorities, isolated replication, control plane extensions

9.1 Scaling the control plane roles

This section explains why control plane roles may need multiple clusters and how that changes ingress, developer self-service, and platform product management.

• Control plane scaling pressures: Explain why a single cluster per control plane role may no longer suffice as an organization grows, and distinguish scaling for performance or locality from scaling for isolation.
  Why it matters: The reason for adding clusters determines what kind of scaling the platform must support.
• Multi-cluster ingress changes: Describe the two ingress changes required when a control plane role spans multiple clusters: routing traffic to clusters and extending developer self-service across them.
  Why it matters: Adding clusters changes both how requests reach services and what developers need from self-service ingress.
• Shared roadmap across isolated platforms: Explain why isolation demands may duplicate an entire platform product definition without creating a separate product roadmap for each line of business.
  Why it matters: Separate platform instances need not become separately defined platform products.

Key terms: control plane role, capacity, locality, isolation, ingress architecture, load balancing, geo-routing, self-serve ingress, service mesh, platform product definition

9.1.1 Dynamic release pipeline

This section explains why growing control-plane roles need dynamically generated release pipelines and where that approach stops meeting scaling needs.

• Static pipeline scaling limits: Explain why a single release pipeline with hard-coded clusters becomes brittle as the number of clusters grows.
  Why it matters: A simple pipeline can become costly to operate when the cluster footprint grows.
• Dynamic release pipeline generation: Describe how a dynamic release pipeline uses a global definition of roles and clusters to generate matching deploy and test steps at run time.
  Why it matters: The release workflow can follow the current cluster inventory without editing its steps for every cluster change.
• Dynamic pipeline tooling choices: Explain why the implementation depends on pipeline-tool capabilities and what alternatives the section suggests when dynamic pipelines are unavailable.
  Why it matters: The chosen pipeline tool must support the scaling approach before the static pipeline becomes a bottleneck.
• Customer isolation scaling needs: Distinguish scaling the number of regions within a role from scaling production clusters for customer or contract isolation, and explain the resulting self-service need.
  Why it matters: The reason clusters are multiplying determines whether pipeline generation alone addresses the scaling problem.

Key terms: static orchestration pipeline, control plane roles, dynamic release pipeline, global definition, deploy and test phases, matrix build, matrix capabilities, dynamic pipelines, cluster provisioning strategies, customer or contract isolation

9.2 Scaling the orchestration of control plane services and extensions

This section explains how Epetech can scale service and extension deployments by moving from cluster-by-cluster pipeline deployments to role-based configuration and local cluster deployment.

• Limits of centralized deployment: Explain why Epetech’s original deploy-and-test pipelines become difficult to operate as clusters and services multiply, and why separate lifecycle pipelines help but do not fully solve the scaling problem.
  Why it matters: It explains why simply adding more pipelines is not enough to scale deployments.
• Role-based distributed deployment: Describe how service and extension pipelines update configuration by cluster role while clusters pull that configuration and deploy and test locally.
  Why it matters: It lets service and cluster counts grow without a proportionate increase in central deployment work.
• Central visibility for local deployments: Explain how the distributed deployment approach retains a unified view of deployment results, monitoring, and alerts.
  Why it matters: Local deployment should not mean losing a platform-wide view of its health.
• GitOps deployer choice: Relate the independent cluster deployment engine to GitOps and describe the limited tool guidance the section gives for Kubernetes clusters.
  Why it matters: It connects the proposed architecture to a recognizable deployment pattern and practical deployer requirements.

Key terms: lifecycle pipeline, services, extensions, clusters, cluster role, central configuration definition, local deployer, health tests, observability tooling, centralized datastore

9.2.1 Scaling the creation of OpenID Connect Assumable Roles

This section explains why creating OpenID Connect assumable roles through an extensions pipeline becomes difficult to scale and proposes an operator-based alternative.

• Scalable assumable role provisioning: Explain why the Terraform resource used in the extensions pipeline is only a starting point for creating OpenID Connect assumable roles, and describe how an administrative operator could provision a role requested by an extension deployment instead.
  Why it matters: Role provisioning needs a manageable path as the number of extensions grows.

Key terms: OpenID Connect assumable roles, extensions pipeline, operator-based provisioner, administrative operator, role resource request

9.3 Scaling through platform event streaming

This section explains why event streaming helps a growing platform scale and illustrates how upgrade events could identify affected deployments and notify their teams.

• Event-driven platform scaling: Explain how the cluster configuration example uses events and why an event-driven architecture helps a platform scale.
  Why it matters: Independent event consumers make it easier to add platform capabilities without tightly coupling every service.
• Upgrade coordination at scale: Explain why growing platform usage creates a coordination problem that cluster configuration scaling alone does not solve.
  Why it matters: Scaling deployments is not enough if user teams cannot prepare for changes that will eventually break them.
• Event-driven upgrade notifications: Trace the proposed path from a Kubernetes upgrade event to actionable notices for affected teams.
  Why it matters: Connecting upgrade detection to team ownership turns a platform change into specific, timely actions for affected users.

Key terms: event-driven architecture, webhook, event-stream publisher, subscriber, decoupling services, platform users, upgrades, deprecation, backward compatibility, apiVersion

9.3.1 release-api

This section uses an Istio rollout to show how lightweight release events can feed a release-api that reacts to changes across a distributed platform.

• Lightweight rollout events: Describe how events can represent the progress of an Istio upgrade across participating Kubernetes clusters, and explain why an event does not always need a pass/fail state.
  Why it matters: Keeping events simple lets teams report rollout progress without forcing every event to represent a final outcome.
• Release API event consumption: Explain the proposed role of a release-api in the rollout example and identify the complication posed by the events’ origins.
  Why it matters: A shared event stream offers a way for platform and application teams to evolve with less direct interference.

Key terms: events, pass/fail, Istio upgrade, release-api, immutable event stream

9.3.2 Adapter pattern

The section explains how adapters standardize release events from different systems so consumers can use a shared event stream.

• Source event adapters: Explain why each type of source event needs an adapter and what the adapter does before the event enters the release API’s stream.
  Why it matters: A common event format lets downstream services work with events regardless of where they originated.
• Decoupled event consumers: Describe how separating event publishing from consumption makes it easier to add systems and build consumers that act on release events.
  Why it matters: Producers and consumers can be added without designing a separate connection for every pairing.
• Consumer starter kits: Explain who can build consumers for the standardized event stream and the role of platform-provided starter kits.
  Why it matters: A standard stream makes event-driven enhancements available beyond the platform team.

Key terms: adapter, standardized event model, release-API, event stream, publishing, consuming, consumer plugins, starter kits, consumer service, application teams

9.3.3 Adapter pattern: Issue tracking

This section illustrates how an issue-tracking consumer turns platform release events into application-team tasks and updates their urgency as production rollout approaches.

• Event-driven issue tracking: Trace how a Jira consumer responds to Kubernetes release events: it creates a medium-urgency issue for an application team when a new version reaches preview, then raises the task’s urgency if verification remains incomplete as production release approaches.
  Why it matters: Release events become actionable work for the teams whose applications need checking.

Key terms: Jira consumer, release-API, preview environment, deprecation events, task urgency

9.3.4 Adapter pattern: CI hooks

This section shows how CI hooks can turn platform upgrade announcements into automated conformance checks, reducing manual coordination while surfacing failures for teams to fix.

• Event-driven upgrade conformance checks: Trace how a CI-hook consumer can respond to a platform component upgrade in preview by triggering application teams’ deployment and conformance tests, then handling passing or failing results.
  Why it matters: The hook turns an upgrade notice into a concrete compatibility check and gives teams useful evidence when something fails.
• Automation benefits and limits: Explain why automating upgrade conformance checks improves platform maintenance velocity, while recognizing what the checks cannot guarantee.
  Why it matters: Automation lets routine, backward-compatible upgrades move faster without pretending that tests catch every problem.

Key terms: consumer API, inbound webhooks, conformance test pipelines, preview environment, Jira ticket, event-driven conformance testing, operational maintenance velocity, backward compatibility

9.3.5 Adapter pattern: Observability hooks

This section shows how an observability adapter publishes release events alongside operational indicators so teams can connect changes to service health.

• Release events in observability: Describe how an observability consumer publishes deployment, rollout-status, and dependency-update events to an observability system alongside normal operational indicators.
  Why it matters: Teams can see delivery changes in the same place they inspect operational health.
• Cross-team change correlation: Explain how shared deployment events help a service team connect an error spike or performance change to an upstream team’s release.
  Why it matters: A team can investigate a dependency’s recent change without relying solely on incident reports or another team’s recollection.
• Automated observability event hooks: Describe the practical route the section gives for emitting deployment-related events and ingesting them with metrics.
  Why it matters: Automatic hooks make the link between delivery activity and operational data consistent across teams.

Key terms: observability consumer, PublishAppDeployed, PublishRolloutStatus, PublishDependencyUpdates, observability indicators, shared observability system, upstream change, downstream error spike, root cause analysis, OpenTelemetry

9.3.6 Adapter pattern: Configuration management database and audit gathering

This section shows how a consumer can forward deployment events to ServiceNow to keep a configuration management database current for audits.

• Deployment events for audit records: Trace how a ServiceNow consumer uses new release events to automate audit gathering and update the configuration management database. Explain why deployment events can supply information the database needs.
  Why it matters: Reusing deployment events can make audit records easier to maintain without collecting the same information again.

Key terms: ServiceNow consumer, release events, deployment event, configuration management database, CMDB, application map

9.4 Summary