Secret management gives an engineering platform one authoritative place for credentials used by its pipelines and services. A secrets store belongs in the initial platform foundation alongside source control, an infrastructure state store, and a pipeline orchestrator. Choose a service that integrates easily with pipelines and supports automated credential rotation. section 6.2 section 6.5
Do not copy every secret into the pipeline tool’s environment-variable service. Keep only the credential needed to access the secrets manager there, and let jobs retrieve other values when needed. This reduces duplicated values and human error, makes rotation easier, and limits the work required to change either the pipeline tool or the secrets manager. section 6.3.3
Bring service-account credentials under automated management in stages:
- Create the service accounts and their roles, then manually generate and store the first credentials. The pipeline cannot fetch credentials that have not yet been stored. section 6.4.4
- Have the pipeline fetch credentials, apply infrastructure changes, and rotate the credentials in AWS and the secrets store. Schedule recurring rotation rather than relying only on infrastructure changes. section 6.4.4
- For application deployment, consider an API that generates, stores, and rotates deployment credentials. A deployment service such as Flux or Argo offers a different pattern that can avoid giving pipelines service accounts with direct Kubernetes API access. section 8.2.1
Figure 6.19 Our service account has two valid credentials at any given time. Only the latest credential is maintained in the secrets store, and the prior keys remain valid until the next rotation to prevent workloads and orchestration from failing due to the race condition of the key being changed while a job is already in process.
The rotation diagram shows why the store holds the newest credential while the previous key remains temporarily valid: an in-flight job must not fail when rotation occurs. section 6.4.4
