Roll-forward is an infrastructure-as-code recovery technique: apply a subsequent corrective change to create a new working state instead of restoring infrastructure to an earlier state. Immutability means every change creates new resources or state, so a true rollback is not available. section 11.0 section 11.1.1
Use a roll-forward approach to restore functionality while limiting further disruption:
- Stabilize the service and restore functionality before pursuing the root cause. section 11.0
- Inspect version-control history, identify the faulty change, and use IaC rather than manual break-glass repairs. section 11.1.1
- Revert the faulty commit when matching the prior working behavior is sufficient, then promote the resulting change through testing and production. section 11.1.1
- Create new environments or services and shift traffic with a canary deployment when reverting does not restore the system. section 11.1.2
- Reconcile drift, dependencies, and environment differences before re-implementing the original change. section 11.4 section 11.3.3
Restoring the promotions service
Removing roles/cloudsql.admin breaks the promotions service. Inspecting Git history reveals the faulty commit; reverting it adds the role back through the pipeline and produces a new state matching the previous working behavior. If that repair fails, deploy a new promotions service and gradually move traffic to it. section 11.1.1 section 11.1.2
Figure 11.2 You add the administrative database role for the promotions service to roll forward the system to a working state.
Shows the administrative database role being added back to restore the promotions service. section 11.1.1
