Chapter 9: Implementing system commands with sagas
The chapter introduces sagas as a service collaboration pattern for commands that update data across multiple services.
After this chapter you will be able to
- Sagas and distributed commands. Explain that a distributed system command updates data in multiple services and that the Saga pattern is a higher-level service collaboration pattern for implementing such commands.
- Cross-service transaction boundaries. Explain why a data-modifying system operation is harder to keep consistent in a microservice architecture than in a monolith.
- System command correctness challenges. Explain why a command such as createOrder() needs transactional behavior: reserving a consumer’s credit and creating an order involve related changes that must remain correct despite failures and concurrent commands.
- Transaction boundary and commit rules. Describe a transaction as a unit of work defining a correctness boundary, including when its changes can commit and how the boundary governs concurrent operations and visibility.
- ACID guarantees for commands. Explain the four ACID properties and why atomicity and isolation are especially important for system commands.
- Two-phase commit coordination. Trace how XA's two-phase commit coordinates participants to commit or roll back together.
- Technology and deployment constraints. Explain why technology support and dynamic deployments make 2PC/XA problematic for microservices.
- Availability cost of coordination. Justify why synchronous coordination reduces availability and why the section favors sagas over distributed ACID transactions.
- Saga transaction sequence. Explain how a saga implements a system command spanning multiple services as a series of local, typically ACID, transactions.
- Saga rollback and isolation. Distinguish sagas from ACID transactions in how committed changes are rolled back and whether the overall operation has isolation.
- Create Order compensation sequence. Trace how the Create Order saga creates an order, reserves credit, and handles a failed credit reservation. Explain why earlier updates need explicit compensation, including the order in which compensations run.
- Pending order semantic lock. Explain why compensation alone does not protect concurrent transactions, and describe how the order’s PENDING, APPROVED, and rejected states are used in the revised saga.
- Pivot transaction and commitment point. Explain why the pivot is the last saga step that can fail because of a business-rule violation, and why its success means the saga can complete without compensating the pivot itself.
- Compensatable steps before the pivot. Describe which steps need compensating transactions and explain why placing such steps before the pivot can create a concurrency problem.
- Retriable steps after the pivot. Explain why steps after the pivot are called retriable and what it means for them to be guaranteed to succeed in the context of the preceding saga steps.
- Saga step and data coordination. Trace how coordination logic invokes saga participants, handles a failed local transaction, and passes data to participants that need it.
- Synchronous response timing tradeoff. Explain the trade-off between responding after a saga completes and returning a partial outcome early, including how the client learns the final result.
- Sequential and parallel saga steps. Distinguish sequential execution from parallel execution of independent saga steps, including the failure-handling cost of parallelism.
- Participant timeout consistency risk. Explain the choice between waiting for a slow participant and timing out, and why a timeout may require a cancellation request.
- Saga runtime and deployment benefits. Explain how sagas differ from classic distributed transactions in runtime coupling and suitability for deployment environments with ephemeral service instances.
- Saga isolation and rollback drawbacks. Explain why sagas require careful design around data consistency and explicit rollback behavior.
- Saga ACD properties and isolation. Distinguish a saga’s atomicity, consistency, and durability from the isolation provided by an ACID transaction. Explain that a saga does not make concurrent execution appear equivalent to some serial order.
- Concurrent saga anomalies. Explain how another request accessing data between saga steps can read inconsistent data or have its update overwritten by compensation. Describe an anomaly as a read or write that would not occur under serial execution.
- Missing global saga isolation. Explain why concurrent sagas can produce an outcome that does not match any serial execution, even when each service correctly handles its local transactions.
- Cancellation during order creation. Trace why cancelling an order while it is being created can leave the Order rejected and the Customer’s available credit reduced, rather than producing the expected cancelled order with unchanged credit.
- Credit exposed by failed cancellation. Explain how cancelling one order while creating another can let an over-limit new order succeed, even though the cancellation ultimately fails.
- Semantic locks and pending states. Explain how a semantic lock signals that a saga may still change a record, and describe how other operations can respond to that lock.
- Commutative update operations. Explain why designing updates to work in either order lets a compensation undo its own change without overwriting another saga’s update.
- Pessimistic saga step ordering. Describe how the pessimistic view countermeasure reorders saga steps to reduce business risk from a dirty read.
- Reread before updating. Explain how rereading a record before an update detects an intervening change, and what the saga does if the record has changed.
What it builds on
- Overview of interprocess communication in a microservice architecture (chapter 8) · Introduces interprocess communication in a microservice architecture.
- Implementing resilient synchronous communication (chapter 8) · Relates to the availability concerns of synchronous communication.
- Interaction styles (chapter 8) · The distinction between synchronous and asynchronous interactions.
- APIs have Service Level Objectives (SLOs) (chapter 8) · APIs and their service-level objectives.
Section by section
Introduction
The chapter introduces sagas as a service collaboration pattern for commands that update data across multiple services.
- Sagas and distributed commands: Explain that a distributed system command updates data in multiple services and that the Saga pattern is a higher-level service collaboration pattern for implementing such commands.
Why it matters: This identifies the cross-service problem that sagas are intended to address.
Key terms: IPC mechanisms, service collaboration patterns, Saga pattern, distributed system commands
9.1 Transaction management in a microservice architecture
Introduces the transaction-management challenge for system commands that modify data across multiple microservices.
- Cross-service transaction boundaries: Explain why a data-modifying system operation is harder to keep consistent in a microservice architecture than in a monolith.
Why it matters: This is the transaction-management problem that motivates the approaches discussed next.
Key terms: system operation, transactional boundaries, consistency
9.1.1 System commands need to be transactional
This section explains why system commands need transaction boundaries, what transactions and ACID guarantee, and why a single-database monolith can apply those guarantees straightforwardly.
- System command correctness challenges: Explain why a command such as createOrder() needs transactional behavior: reserving a consumer’s credit and creating an order involve related changes that must remain correct despite failures and concurrent commands.
Why it matters: These are the correctness problems a transaction must address, not merely steps in implementing an order. - Transaction boundary and commit rules: Describe a transaction as a unit of work defining a correctness boundary, including when its changes can commit and how the boundary governs concurrent operations and visibility.
Why it matters: The transaction boundary specifies what it means for the whole command to succeed correctly. - ACID guarantees for commands: Explain the four ACID properties and why atomicity and isolation are especially important for system commands.
Why it matters: ACID names distinct guarantees, including the two that directly address partial failure and concurrent commands. - Single database transaction wrapping: Describe how a monolithic application with one relational database can execute the order-creation SQL statements with ACID semantics by wrapping them in a transaction, while noting that this is not guaranteed to be as easy in a microservice architecture.
Why it matters: The single-database case establishes the straightforward baseline for the microservice transaction problem.
Key terms: system command, partial failure, concurrency, invariant, availableCredit, transaction, unit of work, transaction boundary, commit, abort
9.1.2 ACID transactions in a microservice architecture
Explains how distributed ACID transactions work and why they are usually a poor fit for system commands that span microservices.
- Local and distributed commands: Distinguish a system command contained within one service from one that spans services, using createOrder() as the example.
Why it matters: This distinction identifies when ordinary transaction management stops being straightforward. - XA participation requirements: Describe what an XA-based distributed transaction requires of its participants and communication.
Why it matters: XA's participation requirements help explain why it constrains a microservice architecture. - Two-phase commit coordination: Trace how XA's two-phase commit coordinates participants to commit or roll back together.
Why it matters: The coordination process explains both the consistency XA provides and the dependencies that make it troublesome. - Technology and deployment constraints: Explain why technology support and dynamic deployments make 2PC/XA problematic for microservices.
Why it matters: These constraints make distributed ACID transactions a poor default for modern microservices. - Availability cost of coordination: Justify why synchronous coordination reduces availability and why the section favors sagas over distributed ACID transactions.
Why it matters: The availability cost is a central reason to look beyond cross-service ACID transactions.
Key terms: system commands, ACID transactions, distributed system commands, createOrder(), XA, global transaction, global transaction identifier, XA-compliant, two-phase commit (2PC), transaction coordinator
9.2 Using the Saga pattern
This section introduces sagas as a way to implement system commands across services using local transactions, and explains how they differ from ACID transactions.
- Saga transaction sequence: Explain how a saga implements a system command spanning multiple services as a series of local, typically ACID, transactions.
Why it matters: This is the basic structure for carrying out one command across services without treating it as one distributed ACID transaction. - Saga step service operations: Describe what a service operation does during a saga step and what it may return.
Why it matters: It connects the saga’s transaction sequence to the business work performed by each service. - Saga rollback and isolation: Distinguish sagas from ACID transactions in how committed changes are rolled back and whether the overall operation has isolation.
Why it matters: Local commits and missing isolation are two constraints a design must account for when using sagas.
Key terms: Saga pattern, system command, local transactions, saga steps, saga step, service operation, business logic, persistent entities, saga participant, compensating transactions
9.2.1 An example saga: The Create Order saga
The section develops the Create Order saga from a happy-path sequence into one that compensates for business-rule failures and prevents other transactions from acting on an unfinished order.
- Create Order compensation sequence: Trace how the Create Order saga creates an order, reserves credit, and handles a failed credit reservation. Explain why earlier updates need explicit compensation, including the order in which compensations run.
Why it matters: It explains how the command can recover from a business-rule failure after another service has already made an update. - Business and technical failures: Distinguish the business-rule failures that cause this saga to compensate and the system command to fail from technical failures that are assumed to delay saga execution.
Why it matters: The distinction clarifies which failures the example’s compensating transaction is meant to handle. - Pending order semantic lock: Explain why compensation alone does not protect concurrent transactions, and describe how the order’s PENDING, APPROVED, and rejected states are used in the revised saga.
Why it matters: Rejecting a failed order is not enough if another transaction can act on it before the rejection occurs.
Key terms: Create Order saga, createOrder(), reserveCredit(), rejectOrder(), compensating transaction, failure, business rule, technical failures, semantic lock, application-level locking
9.2.2 The structure of a saga
This section explains how a saga’s steps are organized around a pivot transaction and why steps before and after it have different rollback and retry requirements.
- Pivot transaction and commitment point: Explain why the pivot is the last saga step that can fail because of a business-rule violation, and why its success means the saga can complete without compensating the pivot itself.
Why it matters: Identifying the pivot tells you where a saga stops needing to undo completed steps. - Compensatable steps before the pivot: Describe which steps need compensating transactions and explain why placing such steps before the pivot can create a concurrency problem.
Why it matters: Compensation can undo a step, but it does not prevent other transactions from seeing that step’s temporary effects. - Retriable steps after the pivot: Explain why steps after the pivot are called retriable and what it means for them to be guaranteed to succeed in the context of the preceding saga steps.
Why it matters: The saga can pass its go/no-go point only if its remaining steps can eventually finish.
Key terms: pivot transaction, go/no-go point, compensating transaction, reserveCredit(), compensatable transactions, forward transaction, retriable transactions, transient technical failure, approveOrder(), PENDING state
9.2.3 Overview of saga coordination
This section explains how saga coordination advances steps and passes data, then examines response timing, parallel execution, timeouts, and two ways to organize coordination.
- Saga step and data coordination: Trace how coordination logic invokes saga participants, handles a failed local transaction, and passes data to participants that need it.
Why it matters: This is the basic work that either form of saga coordination must accomplish. - Synchronous response timing tradeoff: Explain the trade-off between responding after a saga completes and returning a partial outcome early, including how the client learns the final result.
Why it matters: Response timing determines whether the client gets an immediate final answer or must follow up, and how strongly its request depends on the participants. - Sequential and parallel saga steps: Distinguish sequential execution from parallel execution of independent saga steps, including the failure-handling cost of parallelism.
Why it matters: Parallel steps can shorten a saga, but their outcomes make failure handling harder unless the operations are safely independent. - Participant timeout consistency risk: Explain the choice between waiting for a slow participant and timing out, and why a timeout may require a cancellation request.
Why it matters: A timeout limits waiting, but treating silence as proof of failure can leave the system inconsistent. - Choreography and orchestration distinction: Distinguish where decision-making and sequencing reside in choreography versus orchestration.
Why it matters: The distinction identifies who decides and triggers the next saga step.
Key terms: coordination logic, saga participant, local transaction, compensating transactions, final outcome, partial outcome, runtime coupling, 202 (Accepted), 201 (Created), sequentially
9.2.4 Benefits and drawbacks of sagas
This section weighs sagas’ runtime and deployment benefits against their isolation and rollback drawbacks.
- Saga runtime and deployment benefits: Explain how sagas differ from classic distributed transactions in runtime coupling and suitability for deployment environments with ephemeral service instances.
Why it matters: These benefits help explain why sagas fit microservice deployments. - Saga isolation and rollback drawbacks: Explain why sagas require careful design around data consistency and explicit rollback behavior.
Why it matters: These drawbacks explain why using a saga does not remove the need to design for consistency and failure.
Key terms: loose runtime coupling, deployment simplicity, ephemeral service instances, lack of isolation, lack of automatic rollback, compensating transactions
9.3 Handling saga’s lack of isolation
This section explains why sagas lack ACID isolation, how that can cause anomalies, and why reduced isolation can still be workable.
- Saga ACD properties and isolation: Distinguish a saga’s atomicity, consistency, and durability from the isolation provided by an ACID transaction. Explain that a saga does not make concurrent execution appear equivalent to some serial order.
Why it matters: The missing isolation guarantee explains why concurrent requests need special attention in saga-based logic. - Concurrent saga anomalies: Explain how another request accessing data between saga steps can read inconsistent data or have its update overwritten by compensation. Describe an anomaly as a read or write that would not occur under serial execution.
Why it matters: This is the concrete data-consistency risk behind a saga’s lack of isolation. - Practical reduced isolation tradeoff: Explain why lacking full isolation does not automatically make sagas unworkable: developers also accept weaker database isolation for performance, and saga countermeasures can address the resulting risk at a high level.
Why it matters: It frames saga isolation as a risk to manage, not a reason to dismiss sagas outright.
Key terms: ACID, isolation, ACD, atomicity, consistency, durability, anomaly, compensating transaction, serially, reduced isolation
9.3.1 Overview of anomalies
This section explains how concurrent sagas can violate business invariants despite correct local transactions, using two order-processing examples.
- Missing global saga isolation: Explain why concurrent sagas can produce an outcome that does not match any serial execution, even when each service correctly handles its local transactions.
Why it matters: Correct local transactions are not enough to guarantee a correct outcome for a system command. - Read anomalies during sagas: Distinguish dirty reads, nonrepeatable reads, and phantom reads by what a saga step observes and what another transaction or saga changed.
Why it matters: These distinctions help identify what a saga observed while other work was in progress. - Lost updates and write skew: Distinguish a lost update from write skew in concurrent sagas.
Why it matters: The two anomalies can harm different parts of an order workflow at the same time. - Cancellation during order creation: Trace why cancelling an order while it is being created can leave the Order rejected and the Customer’s available credit reduced, rather than producing the expected cancelled order with unchanged credit.
Why it matters: The example shows how interleaving can break the relationship between an order’s state and the customer’s credit. - Credit exposed by failed cancellation: Explain how cancelling one order while creating another can let an over-limit new order succeed, even though the cancellation ultimately fails.
Why it matters: Temporary changes made by a saga can wrongly authorize another business operation.
Key terms: isolation, serialization anomalies, global isolation, dirty reads, nonrepeatable reads, phantom reads, lost updates, write skew, business invariant, Create Order Saga
9.3.2 Countermeasures for handling the lack of isolation
The section explains six countermeasures for preventing or reducing business problems caused by sagas’ lack of transaction isolation.
- Semantic locks and pending states: Explain how a semantic lock signals that a saga may still change a record, and describe how other operations can respond to that lock.
Why it matters: A pending state helps other sagas avoid treating an unfinished update as final. - Commutative update operations: Explain why designing updates to work in either order lets a compensation undo its own change without overwriting another saga’s update.
Why it matters: Order-independent updates make compensation safer when sagas run concurrently. - Pessimistic saga step ordering: Describe how the pessimistic view countermeasure reorders saga steps to reduce business risk from a dirty read.
Why it matters: Step order can keep another saga from acting on a credit increase that is not yet safe to rely on. - Reread before updating: Explain how rereading a record before an update detects an intervening change, and what the saga does if the record has changed.
Why it matters: A final check can stop an approval from overwriting a concurrent cancellation. - Recorded out of order operations: Explain how a version file records operations so a service can handle requests that arrive in the wrong order.
Why it matters: Recording request history prevents an out-of-order authorization from undoing the intended cancellation. - Risk based concurrency selection: Describe how the by value countermeasure chooses a concurrency mechanism for each request based on business risk, and state the section’s qualification about its suitability.
Why it matters: The option illustrates a business-risk trade-off, even though the author advises against it for modern microservices.
Key terms: semantic lock, *_PENDING states, APPROVAL_PENDING, compensatable transaction, retriable transaction, compensating transaction, commutative updates, lost updates, debit(), credit()
9.4 Summary
Where this is used later
- The choreography-based version of the Create Order Saga (chapter 10) · The choreography-based Create Order saga builds on the command’s forward and compensating steps.
- Benefits and drawbacks of choreography-based saga coordination (chapter 10) · Introduces the Create Order Saga used in the example.
- The FTGO Create Order Saga definition (chapter 11) · The FTGO Create Order Saga definition applies the structure of saga steps.
- Choreography-based sagas (chapter 10) · Choreography distributes the coordination work among participants.
- Orchestration-based sagas (chapter 10) · Orchestration centralizes the coordination work.
- (chapter intro) (chapter 10) · Introduces saga coordination.
- Design elements: events, channels, publishers and handlers (chapter 10) · the overview of saga coordination
- Why system queries need to be transactional (chapter 12) · The significance of isolation when transactions execute concurrently.
Concepts explained in this chapter
- sagaA distributed transaction pattern that coordinates a sequence of local transactions and, when necessary, compensating actions across services.
- isolationA database transaction setting that determines which effects of concurrent transactions may be observed.
- commutative updateA commutative update is an update operation designed to produce an equivalent result regardless of the order in which compatible updates execute.
- pessimistic viewA pessimistic view is a saga design technique that orders steps to reduce business risk from observing intermediate or dirty data.
- reread valueReread value is a concurrency technique that rereads a record before overwriting it and aborts when the record has changed since its earlier read.
- semantic lockA semantic lock is an application-level record flag or locking convention that indicates an uncommitted value may change and should be restricted or treated cautiously.
- version fileA version file is a record of operations on a database record that allows those operations to be reordered during concurrent processing.
- commandA system operation that mutates application data.
- invariantAn invariant is a condition that must remain true for application data or system state to be valid.
- retriable transactionA retriable transaction is a Saga step after the pivot transaction that is guaranteed to succeed, potentially through repeated invocation after transient failure.
- ACIDA database transaction that provides atomicity, consistency, isolation, and durability guarantees.
- compensating transactionA transaction that semantically undoes the effects of a previously committed transaction.
- distributed transactionA transaction identified and coordinated across multiple participating resource managers.
- dirty readA dirty read is a concurrency anomaly in which a transaction reads data written by another transaction before that write is committed.
- lost update anomalyA concurrency anomaly in which one update overwrites another update made from the same previously read data without detecting the conflict.
- compensatable stepA saga step before the pivot transaction whose effects may need to be undone by a compensating transaction.
- pivot transactionA saga transaction that marks the point after which the saga must complete through forward recovery or subsequent noncompensatable actions.
- atomicityAtomicity is the transaction property that all operations succeed as one unit or the transaction has no effect.
- durabilityThe transaction property that preserves committed changes despite subsequent failures or restarts.
- ACDACD is a transaction model providing atomicity, consistency, and durability without isolation.
- by valueBy value is a concurrency strategy that selects transaction mechanisms according to the business risk of each request.
- pending stateA pending state is a business-entity state indicating that an operation is in progress and its result is not yet committed.
- choreography-based sagaLinear choreography is a saga coordination pattern in which events trigger saga participants sequentially.
- orchestration-based saga coordinationA saga whose steps are coordinated by a centralized orchestrator.
- parallelismParallel execution runs independent saga steps concurrently rather than waiting for each step to finish before starting the next.
- partial outcomeA partial outcome is a result that reports acceptance or intermediate progress before the complete processing outcome is known.
- saga coordination logicSaga coordination logic is the control logic that sequences saga steps, invokes participants, transfers data, and triggers compensating transactions.
- saga orchestratorA saga orchestrator coordinates saga steps by invoking participants, maintaining saga state, and determining what to do based on their outcomes.
- saga participantA service or component that performs one or more local transactions as part of a saga.
- sequential processingSequential execution runs saga steps one after another, starting each step after the preceding step completes.
- service-level objectiveA target value for a service-level indicator that expresses a desired level of reliability or performance.
- timeout mechanismA timeout mechanism limits how long an operation may wait for a response before treating the wait as a failure or exceptional condition.
- consistencyConsistency is the transaction property that a committed transaction leaves the system in a valid state while preserving defined invariants and constraints.
- countermeasureA saga countermeasure is an application-level design technique that reduces anomalies caused by the absence of isolation across saga transactions.
- transaction boundaryA transaction boundary defines the scope within which operations are governed by rules for success, failure, concurrency, conflicts, and visibility.
- serialization anomalyA concurrency anomaly in which the combined outcome of interleaved transactions or sagas is not equivalent to any serial execution.
- write skewA concurrency anomaly in which concurrent operations read overlapping data and update disjoint data in a way that violates an invariant.
- concurrencyConcurrency is the condition in which multiple operations or transactions execute during overlapping periods of time.
- database transactionA unit of work whose operations execute under defined consistency and completion semantics.
- partial failureA distributed-system failure in which an invoked service is unavailable or responds with excessive latency while other components continue operating.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
