Chapter 2: The layer nobody designed
This introduction frames Lumen Retail’s promotion layer as the unintended result of reasonable engineering decisions, rather than careless work.
After this chapter you will be able to
- Origins of the improvised layer. Explain why Lumen Retail’s promotion layer should not be dismissed as the result of engineers cutting corners, and describe what their accumulated decisions produced.
- Limits of branch merging. Explain why merging a pre-prod environment branch into a production branch would not safely promote only the image, while distinguishing that objection from an objection to environment branches themselves.
- Selective overlay image update. Describe how Lumen represents its environments in the GitOps repository and what it changes to move a build to production.
- Promotion decision versus edit. Explain why automating the file edit is easier than deciding when an image is safe to advance.
- Hand-triggered CI promotion workflow. Trace the workflow Lumen builds with GitHub Actions, including where people still make and approve the production decision.
- Production test gate. Explain how an approved production promotion carried an untested change, and how Checkout added a pre-prod test gate.
- Soak eligibility and polling. Describe why Checkout added a soak requirement and how that requirement changed promotion from a manual trigger to a scheduled workflow.
- Copied workflow maintenance. Explain how Checkout’s successful workflow spread to other teams and why maintaining a shared fix became difficult.
- Version-blind test results. Trace how checking only the latest completed test run could allow an untested version to advance.
- Scale of the improvised layer. Explain the two ways Lumen scaled—more teams using the workflow and more environments per service—and why its original design became costly to rely on.
- Branching environment progression. Trace how one build moves from dev through parallel checks and regional environments, and explain why its progress cannot be described by a single position.
- Preexisting gaps in job-built promotion. Explain that the gaps in Checkout’s promotion layer existed even when its two-environment setup worked well; scale made those gaps costly rather than creating them.
- Deployment state versus version progress. Distinguish the deployment question Argo CD answers from the version-progress question Lumen cannot answer directly. Explain why answering the latter requires assembling facts from separate places.
- Promotion eligibility between CI runs. Explain why a CI job’s start-and-stop execution does not fit promotion decisions, and how Checkout’s polling job compensates.
- Team variation and duplicated machinery. Distinguish legitimate differences in teams’ production requirements from the duplicated machinery used to check those requirements. Explain how separate copies create recurring maintenance work and make claims about a version’s readiness less reliable.
- Shared workflow versus control plane. Explain what centralizing code in a shared workflow fixes and why it does not provide a complete promotion solution. Describe the additional ongoing awareness a promotion control plane would need and the cost of building one in-house.
- Promotion controls and audit evidence. Explain Lumen’s two promotion controls and why the auditor examines approval evidence differently from an automated testing gate.
- Fragmented and faulty testing gates. Explain why Lumen cannot reliably substantiate its pre-prod testing control, and why assembling audit evidence across teams is hard to verify.
- Continuous promotion control plane. Distinguish a promotion control plane from a collection of jobs, using the two-hour pre-prod soak as an example.
- Queryable version position. Explain how the control plane makes a version’s progress across environments visible.
- Shared promotion machinery. Explain what becomes shared across teams and what decisions remain with each team.
- Built-in promotion evidence. Describe how an advance produces an audit record and how that record answers an auditor’s questions.
What it builds on
- The improvised layer in modern delivery (chapter 1) · The idea of an improvised layer in modern delivery.
- Queryable lineage (chapter 1) · The earlier discussion of queryable lineage.
Section by section
Introduction
This introduction frames Lumen Retail’s promotion layer as the unintended result of reasonable engineering decisions, rather than careless work.
- Origins of the improvised layer: Explain why Lumen Retail’s promotion layer should not be dismissed as the result of engineers cutting corners, and describe what their accumulated decisions produced.
Why it matters: It explains why the problem can arise even when individual engineering decisions make sense.
Key terms: Lumen Retail, improvised promotion layer, intentional design
2.1 How Lumen built what it has
Lumen’s early promotion workflow shows why moving an image from pre-prod to production requires a selective configuration change and a decision that Git alone cannot make.
- Lumen’s starting delivery setup: Describe how CI updates pre-prod and how Argo CD keeps each environment aligned with its declared state.
Why it matters: This starting point explains why getting an image into pre-prod does not also get it into production. - Limits of branch merging: Explain why merging a pre-prod environment branch into a production branch would not safely promote only the image, while distinguishing that objection from an objection to environment branches themselves.
Why it matters: The problem is not Git branches themselves, but treating a selective promotion as a wholesale merge. - Selective overlay image update: Describe how Lumen represents its environments in the GitOps repository and what it changes to move a build to production.
Why it matters: A targeted image edit lets production keep the settings that intentionally differ from pre-prod. - Promotion decision versus edit: Explain why automating the file edit is easier than deciding when an image is safe to advance.
Why it matters: Automating a Git edit does not, by itself, automate a trustworthy promotion decision. - Hand-triggered CI promotion workflow: Trace the workflow Lumen builds with GitHub Actions, including where people still make and approve the production decision.
Why it matters: This explains how Lumen automated the repetitive change without automating the judgment about when to promote.
Key terms: CI, pre-prod, production, Argo CD Application, desired state, environment branches, merge, artifact references, rendered manifests, GitOps repository
2.2 What success cost
Checkout’s promotion safeguards improve releases, but scheduling and copying those safeguards across teams expose gaps in how the workflows track eligibility and test results.
- Production test gate: Explain how an approved production promotion carried an untested change, and how Checkout added a pre-prod test gate.
Why it matters: An authorized, reviewed change can still reach production before its tests finish. - Soak eligibility and polling: Describe why Checkout added a soak requirement and how that requirement changed promotion from a manual trigger to a scheduled workflow.
Why it matters: A time-based safeguard also creates a need to notice when a version becomes eligible. - Copied workflow maintenance: Explain how Checkout’s successful workflow spread to other teams and why maintaining a shared fix became difficult.
Why it matters: A useful workflow can become expensive to correct when each team owns a different descendant. - Version-blind test results: Trace how checking only the latest completed test run could allow an untested version to advance.
Why it matters: A passing test result is not a reliable promotion gate unless it applies to the version being promoted.
Key terms: pre-prod test suite, test gate, production promotion workflow, soak, eligible, scheduled, polling, Argo CD, workflow, copies
2.3 What scale asked of it
The section shows how Lumen’s growing number of teams and environments exposed the limits of an improvised promotion layer, especially once a service’s path became a branching graph rather than a line.
- Scale of the improvised layer: Explain the two ways Lumen scaled—more teams using the workflow and more environments per service—and why its original design became costly to rely on.
Why it matters: The original workflow’s limitations became consequential as more teams and environments depended on it. - Branching environment progression: Trace how one build moves from dev through parallel checks and regional environments, and explain why its progress cannot be described by a single position.
Why it matters: A branching path makes it necessary to track where a version stands in several environments at once.
Key terms: improvised layer, environments, teams, dev, smoke, perf, pre-prod, production, environment graph
2.4 What the layer never had
The section frames the coming gaps as limitations of Checkout’s job-built promotion layer that existed before scale made them costly.
- Preexisting gaps in job-built promotion: Explain that the gaps in Checkout’s promotion layer existed even when its two-environment setup worked well; scale made those gaps costly rather than creating them.
Why it matters: This distinguishes an underlying limitation of the job-built layer from a problem caused by growth.
Key terms: gaps, two-environment layer, jobs, scale
2.4.1 No view of the whole
This section explains why Lumen can see what an individual environment is running but cannot readily see a version’s progress across its promotion path.
- Deployment state versus version progress: Distinguish the deployment question Argo CD answers from the version-progress question Lumen cannot answer directly. Explain why answering the latter requires assembling facts from separate places.
Why it matters: Seeing a deployment’s current state does not tell a team how far a fix has traveled. - Incident-wide promotion visibility: Explain why the missing whole-path view becomes urgent when a version misbehaves in one production region while heading toward others.
Why it matters: The visibility gap makes it harder to judge where a problematic version has already gone and where it may go next.
Key terms: Argo CD, deployment question, reconstruction, shape of the graph, where each version is within it
2.4.2 Nothing is watching when nothing is running
This section explains why finite CI jobs cannot continuously watch for changes in promotion eligibility, leaving Checkout’s polling job to act as an improvised control loop.
- Promotion eligibility between CI runs: Explain why a CI job’s start-and-stop execution does not fit promotion decisions, and how Checkout’s polling job compensates.
Why it matters: It shows why a workflow suited to building and testing becomes awkward when it must decide when software is ready to advance.
Key terms: CI job, promotion, eligibility, soak, polling job, control loop
2.4.3 Nothing shared between teams
The section explains why legitimate team-specific promotion requirements do not justify duplicating promotion machinery, and why a shared workflow only partly solves the problem.
- Team variation and duplicated machinery: Distinguish legitimate differences in teams’ production requirements from the duplicated machinery used to check those requirements. Explain how separate copies create recurring maintenance work and make claims about a version’s readiness less reliable.
Why it matters: The distinction shows what Lumen needs to share without forcing every team to use identical production checks. - Shared workflow versus control plane: Explain what centralizing code in a shared workflow fixes and why it does not provide a complete promotion solution. Describe the additional ongoing awareness a promotion control plane would need and the cost of building one in-house.
Why it matters: Sharing a job reduces some repeated work, but it does not give Lumen a continuous view of promotion across environments.
Key terms: machinery underneath, health, test result, version, soak, shared workflow, job, promotion control plane
2.4.4 Nothing an auditor can read
The section shows why Lumen can reconstruct approval evidence for a SOC 2 Type II audit but cannot reliably demonstrate that its varied promotion gates enforced the promised testing control.
- Promotion controls and audit evidence: Explain Lumen’s two promotion controls and why the auditor examines approval evidence differently from an automated testing gate.
Why it matters: The two controls require different kinds of evidence, so a list of approved promotions cannot establish that the testing gate worked. - Reconstructable approval trail: Trace how Lumen can assemble a production promotion list and identify the artifact and approver for a sampled promotion.
Why it matters: Lumen’s audit problem is not that all promotion evidence is missing; some of it can be pieced together from Git and pull requests. - Fragmented and faulty testing gates: Explain why Lumen cannot reliably substantiate its pre-prod testing control, and why assembling audit evidence across teams is hard to verify.
Why it matters: Approval records cannot compensate for a gate that checks the wrong version, and fragmented implementations make a complete audit answer difficult to verify.
Key terms: SOC 2 Type II audit, observation period, system of record, automated control, gate, production manifests directory, path-filtered git log, artifact, approver, pull request
2.5 A different kind of system
Devon’s pilot shows how a continuously running promotion control plane replaces Lumen’s scattered jobs with queryable version state, shared machinery, promotion records, and a single governance surface.
- Continuous promotion control plane: Distinguish a promotion control plane from a collection of jobs, using the two-hour pre-prod soak as an example.
Why it matters: The distinction explains why promotion rules need not depend on jobs waking up to check them. - Queryable version position: Explain how the control plane makes a version’s progress across environments visible.
Why it matters: A team can answer how far a version has reached without reconstructing its path from separate workflows. - Shared promotion machinery: Explain what becomes shared across teams and what decisions remain with each team.
Why it matters: One reliable implementation replaces eleven copies without forcing eleven teams to adopt identical release criteria. - Built-in promotion evidence: Describe how an advance produces an audit record and how that record answers an auditor’s questions.
Why it matters: The next audit can start from promotion records rather than a list assembled afterward from commits. - Audit record retention limit: Explain why keeping an audit period’s worth of promotion records may require storage outside the cluster.
Why it matters: Built-in records still need an appropriate retention plan for a full audit period. - Unified promotion governance surface: Explain the organizational benefit of giving promotion its own system at Lumen’s scale.
Why it matters: At Lumen, a designed promotion system gives compliance a coherent place to engage with work that already spans many teams.
Key terms: promotion control plane, controller, soak, stage, environments as a graph, version’s own recorded state, verification result, version, verified in pre-prod, advance
2.6 Summary
Where this is used later
- What Kargo is and where it fits (chapter 3) · The difficulty of reconstructing promotion decisions from the existing delivery layer.
- Stage: a node in the promotion graph (chapter 3) · introduces the Stage resource behind positions in the promotion graph
- Promotion: a durable advancement event (chapter 3) · introduces Promotion as a durable advancement event
Concepts explained in this chapter
- promotion control planeAn architectural layer that coordinates artifact advancement between environments independently of build and runtime synchronization.
- progression graphA directed graph representing deployment environments as stages and the governed paths by which artifacts advance between them.
- promotion workflowAn automated sequence of steps that governs advancing an artifact between deployment environments.
- Kubernetes controllerA control loop that observes Kubernetes resources and acts to move their actual state toward the declared desired state.
- promotion recordA promotion record is durable evidence of an artifact's advancement between deployment environments and the decisions associated with that advancement.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
