Idempotency makes repeated operations produce the same effective result as one execution, protecting ingestion and maintenance workflows from retry-related duplication or corruption. section 11.1.1
Apply the principle by managing execution boundaries and retry behavior:
- Treat batch pipelines as idempotent only when input state and snapshot boundaries are explicitly defined. section 6.2
- Use tracked input offsets, state, and commit boundaries to make micro-batch recovery easier to reason about. section 6.2
- Design orchestration workflows with standardized policies, clear inputs and outputs, and observable side effects. section 11.1.1
Retrying compaction
If a compaction job runs twice after a retry or scheduler glitch, an idempotent design leaves the result equivalent to a single run. section 11.1.1
