Chapter 1: Thinking in distributed systems: Models, mindsets, and mechanics

Introduction

The chapter introduces distributed systems as interacting components and explains why engineers need them to build applications that remain functional under load and failure.

• Distributed Components and Emergent Behavior: Describe a distributed system as collaborating, concurrent components that exchange messages over a network, and explain that its behavior and complexity arise from the components and their interactions.
  Why it matters: Looking at components alone cannot explain how the whole system behaves.
• Reasons for Distributing Applications: Explain why applications use multiple components despite the difficulty: a single component cannot handle unbounded load or survive inevitable failures, while a system must remain functional as load grows and failures occur.
  Why it matters: The need to handle load and failure explains why distributed systems are worth their complexity.

Key terms: distributed system, components, messages, network, emerges, correctness, scalability, reliability, load, failures

1.1 Software engineering and mental models

This section frames accurate, concise mental models as a way to replace confusion with confident reasoning about distributed systems, and distinguishes knowing facts from understanding how to use them.

• Purpose of mental models: Explain why the section calls for accurate, concise mental models: they give engineers an internal representation they can use to reason confidently about a system rather than repeatedly guessing from its behavior.
  Why it matters: A dependable mental model makes a complex system easier to reason about and explain.
• Knowledge versus understanding: Distinguish knowing facts or rules about a distributed system from understanding it well enough to reason about it, as illustrated by the difference between learning chess rules and grasping chess strategy.
  Why it matters: Collecting facts alone does not give you the dependable understanding the book aims to build.

Key terms: mental model, accurate, concise, reason confidently, knowing, understanding, dependable mental models

1.1.1 Mental models: The foundation of reasoning

Defines mental models as internal representations used to understand and communicate about systems, then distinguishes a model’s correctness from its completeness.

• Mental models and ground truth: Explain that a mental model is an internal representation of a target system that supports understanding, communication, and reasoning; distinguish the system’s facts from the facts a person believes about it.
  Why it matters: Separating the system from your picture of it helps you reason about where misunderstandings come from.
• Correctness and completeness criteria: Distinguish correctness from completeness in a mental model: correctness means every fact in the model is true of the system, while completeness means the model includes every relevant fact of the system.
  Why it matters: A model can mislead you either by including something false or by leaving out something relevant.

Key terms: model, mental model, target system, ground truth, perceived truth, correctness, completeness, falsehoods, omissions, relevant facts

1.1.2 Correct mental models

Defines a correct mental model as one whose stated facts match the system it represents.

• Correct mental model criterion: Explain that a mental model is correct when every fact it states is true of the system it represents. Illustrate this with the light bulb: a model stating its actual resistance of 3.0 ohms is correct, while one stating 2.0 ohms is not.
  Why it matters: This gives you a concrete way to check whether a model misrepresents a system.

Key terms: correct mental model, fact of the model, fact of the system

1.1.3 Complete mental models

Completeness means a mental model includes all facts relevant to the question being asked, not every fact about the system.

• Question-dependent model completeness: Explain why a mental model can be complete for one question but incomplete for another: it must capture every relevant fact, and relevance depends on the application. Use the light-bulb example to distinguish predicting current from predicting brightness.
  Why it matters: A model can be correct yet still leave out something needed to answer your particular question.

Key terms: completeness, relevance, luminous efficiency

1.2 Mental model of software systems

The section explains why software systems are understood through virtual models and why analogies used to build those models need careful limits.

• Virtual software system models: Explain that components and network messages are virtual constructs used to model a distributed software system, not physical objects one can find. Describe how different models can be equally valid when they account for the same observable behavior.
  Why it matters: This lets you compare models by what they explain rather than assuming one picture is the physical system itself.
• Limits of teaching analogies: Explain how an analogy can make an unfamiliar concept easier to reason about while also causing misconceptions when its differences from the target concept are overlooked. Describe why its limitations must be assessed before drawing conclusions from it.
  Why it matters: Checking where an analogy stops fitting prevents a helpful comparison from becoming a misleading model.

Key terms: virtual constructs, observable behavior, equivalent models, analogies, mental models, limitations

1.3 Different types of models

This section explains why distributed systems can be described through many models and why comparing those models improves understanding.

• Multiple ways to model systems: Describe how distributed systems can be represented in many ways, including informal or formal descriptions and different machine- or process-based models, rather than by one definitive model.
  Why it matters: Different descriptions need not be competing definitions of the same system.
• Value of multiple mental models: Explain that diverse mental models add complexity but each offers a perspective; studying several can build a more holistic understanding and reveal omissions or inconsistencies in one's thinking.
  Why it matters: Comparing perspectives can reveal gaps that are hard to notice from just one model.

Key terms: mental model, informally, formally, action machines, state machines, state-action machines, unique perspective, holistic understanding, omissions, inconsistencies

1.3.1 Different models describing the same aspects

The section shows that two models can describe the same distributed-system behavior even when they place in-flight messages in different parts of the model.

• Equivalent message buffer models: Explain how one model treats the network as both a communication bus and a buffer for in-flight messages, while another treats it only as a bus and puts those buffers in the components. Describe why the models are equivalent: both can represent messages in flight, lost, duplicated, or reordered.
  Why it matters: A different way of drawing or describing a system does not necessarily mean it captures different behavior.

Key terms: equivalent, communication bus, buffer, in-flight messages

1.3.2 Different models describing different aspects of a system

This section shows that different models of a distributed system may focus on different aspects, and that comparing their purposes gives a fuller understanding.

• Models with different focuses: Explain that models of the same distributed system need not be equivalent: each may include facts another deliberately omits. Contrast the Transaction Commit model, which ignores message exchange, with the Two-Phase Commit model, which accounts for it and can represent lost, duplicated, or rearranged messages.
  Why it matters: A model can be useful without representing every behavior of the system.
• Value of multiple perspectives: Explain how considering multiple models can reveal gaps or misconceptions in your understanding. Describe why, when reading another author's model, you should ask what point it is meant to convey rather than assume that a different model is incorrect.
  Why it matters: Comparing perspectives helps you understand both the system and why an author chose a particular model.

Key terms: Transaction Commit protocol, Two-Phase Commit protocol, message exchange, mental models, multiple perspectives, holistic understanding

1.4 Thinking about distributed systems

Introduces a mental model of a distributed system as message-communicating components whose behavior can be described as a sequence of state-changing steps.

• Components and local state: Describe a distributed system as concurrent, collaborating components that send and receive messages over a network. Explain that each component alone accesses its local state, while the network has its own local state, including messages in flight.
  Why it matters: This identifies what can communicate and what state each part of the system can access.
• State machine behavior: Explain how the section represents system behavior as a sequence of states connected by steps. State that, in this model, each discrete step is completed by exactly one component or the network.
  Why it matters: The state-machine view gives a concise way to reason about how system behavior unfolds.
• External and internal steps: Distinguish external steps, such as sending or receiving a message, from internal steps, such as local computation or accessing local state.
  Why it matters: The distinction makes it easier to describe whether a step involves communication or local activity.

Key terms: component, local state, network, messages in flight, state machine, state action machine, state, step, external steps, internal steps

1.4.1 Correctness

The section defines correctness through safety and liveness, then uses a two-participant distributed transaction to show how those guarantees constrain possible behavior.

• Safety and liveness guarantees: Distinguish safety, which rules out something bad happening, from liveness, which requires something good to eventually happen; explain that a system is correct only if every possible behavior satisfies its guarantees.
  Why it matters: These guarantees turn a vague claim that a system works into criteria for evaluating its behavior.
• Two participant transaction outcomes: Apply safety and liveness to a transaction spanning P1 and P2: neither may commit while the other aborts, and both must eventually decide to commit or abort, so they ultimately reach the same decision.
  Why it matters: The example shows how abstract guarantees identify both inconsistent outcomes and transactions that get stuck.
• Transaction status model: Describe how the example models the transaction by tracking each participant as working, committed, or aborted, with a transition changing one participant at a time; explain that it deliberately leaves out keys, values, and locks because they are not relevant to this discussion.
  Why it matters: Choosing the aspects to model makes the behavior space manageable while keeping the guarantees in focus.

Key terms: correctness, safety, liveness, possible behavior, distributed transaction, participants, P1, P2, committed, aborted

1.4.2 Scalability and reliability

The section defines scalability and reliability through responsiveness and explains why this framing is useful alongside safety and liveness.

• Scalability and reliability guarantees: Distinguish scalability as a system’s ability to remain responsive under load from reliability as its ability to remain responsive when failures occur.
  Why it matters: The distinction identifies the different conditions under which a system must meet its objectives.
• Responsiveness as a mental model: Explain that the author informally defines responsiveness as meeting service-level objectives and prefers this framing for scalability and reliability over describing those guarantees in terms of safety and liveness.
  Why it matters: Different mental models can describe system properties, but one framing may make a particular question easier to think about.

Key terms: scalability, reliability, load, failure, responsiveness, service-level objectives, safety, liveness, mental models

1.4.3 Responsiveness

The section defines responsiveness through service-level measurements and explains why acceptable guarantees depend on the application and emerge from component interactions.

• Responsiveness measurement chain: Explain how a service-level indicator, service-level objective, error rate, and error budget combine to define responsiveness.
  Why it matters: These terms turn a vague claim that a system is responsive into a measurable one.
• Application-specific guarantees: Explain that engineers define their system’s correctness guarantees, including scalability and reliability, according to which behaviors their application considers desirable, tolerable, or intolerable.
  Why it matters: The same behavior can be acceptable for one application and broken for another.
• Emergent system guarantees: Explain why correctness guarantees, including scalability and reliability, arise from components and their interactions rather than from an individual component.
  Why it matters: Assessing a component alone cannot establish how the whole system handles load or failures.

Key terms: service-level indicator, service-level objective, error rate, error budget, responsiveness, application-specific, correctness guarantees, scalability, reliability, emergent properties

1.5 Two big ideas

Introduces two ideas for thinking about distributed systems that will later ground definitions of services and microservices.

• Two foundational viewpoints: State that systems of systems and global versus local viewpoints are two important ideas for thinking about distributed systems, and that they will underpin later definitions of services and microservices.
  Why it matters: These ideas provide the framing for the book’s later discussion of services.

Key terms: systems of systems, global versus local viewpoints, services, microservices

1.5.1 Systems of systems

This section uses holons and holarchies to show how the same distributed system can be understood at different levels of abstraction.

• Components as holons: Explain why a component in a distributed system need not be indivisible: interacting components can form a subsystem that behaves like one component at a higher level. Describe a holon as both a whole and part of a larger whole, and a holarchy as an organization of holons.
  Why it matters: This gives you a way to reason about a subsystem without losing sight of its internal parts.
• Perspective-dependent system boundaries: Explain that the analyst chooses which components to treat as atomic or higher-order entities according to the question at hand. Illustrate this with the same database system viewed as one cluster, three interacting nodes, or two tenant databases whose behavior involves those nodes.
  Why it matters: Changing the grouping helps you describe the relevant behavior of a complex system without requiring one fixed hierarchy.

Key terms: component, subsystem, holon, holarchy, higher-order entity, atomic entity, level of observation, database cluster, database nodes, tenant

1.5.2 Global view vs. local view

The section contrasts an observer’s global view with a component’s limited local view and explains why component-to-component communication depends on the network.

• Global and local views: Distinguish what an all-knowing observer can see from what one component can see, and explain how a component could determine the global system state.
  Why it matters: A component must reason and act without the complete view available to an outside observer.
• Network-mediated message exchange: Explain why a message addressed from C1 to C2 is not direct communication between them: C1 sends it to the network, which may or may not forward it.
  Why it matters: Addressing a message to another component does not mean that component receives it.

Key terms: global view, local view, system state, local state, component, network, message

1.6 Distributed Systems Incorporated

The section uses an office-building analogy to make distributed-system components, communication failures, and uncertain request outcomes easier to reason about.

• Office building system mapping: Explain how rooms and occupants represent independent components with local state and tasks, pneumatic tubes represent the network, and the mailbox represents communication with the outside world. Describe how the same mapping can make a real system, such as Kubernetes or Kafka, more concrete.
  Why it matters: The analogy gives you a concrete way to reason about an otherwise abstract system.
• Office failures and disrupted messages: Describe how employee absences stand for transient, intermittent, and permanent component failures, while lost, duplicated, or rearranged mail stands for messages that are not delivered, delivered more than once, or delivered out of order.
  Why it matters: These concrete disruptions help you anticipate ways components and communication can fail.
• Unanswered request and retry risk: Explain why receiving no response does not tell a sender whether its request was lost or was processed and its response was lost, and why sending the request again could process it twice and cause inconsistencies.
  Why it matters: A missing response creates uncertainty that a simple retry may make worse.

Key terms: Distributed Systems Incorporated, concurrent component, local state, network, external interface, crash semantics, transient failure, intermittent failure, permanent failure, message-delivery semantics

1.7 Navigating complexity

This section introduces informal insights about distributed-system complexity that later chapters will explore in more depth.

• Informal complexity preview: Explain that the section previews a few insights about distributed-system complexity without exploring them deeply yet, and says later chapters will develop them further.
  Why it matters: It sets expectations for an informal preview rather than a complete explanation.

Key terms: aha! moments, complexities of distributed systems

1.7.1 Simple yet complex

Simple rules for individual members of Distributed Systems Incorporated can combine to produce complex behavior across the organization.

• Complexity from simple components: Explain that even when each staff member has simple tasks and rules, their combined behavior can be complex; putting simple components together does not guarantee a simple system.
  Why it matters: It cautions against assuming a system is easy to understand just because its individual parts are.

Key terms: simple components, complex systems, Distributed Systems Incorporated

1.7.2 Emergent behavior

The section explains how a company’s scalability and reliability emerge from employees working together rather than from any one employee.

• Emergent scalability and reliability: Explain why Distributed Systems Incorporated’s scalability and reliability cannot be attributed to one employee, and how they arise from employees’ actions and interactions.
  Why it matters: This is the foundation for understanding how a reliable system can be built from unreliable components.

Key terms: emergent, scalability, reliability, unreliable components

1.7.3 Changing perspective

Distinguishes changes in model detail from changes in observer perspective, then uses a component’s limited view to state a core challenge of distributed systems.

• Resolution versus perspective: Distinguish switching between black-box and white-box views from switching between global and local perspectives. Explain that both black-box and white-box views can depict what an all-knowing observer sees.
  Why it matters: More detail about a system does not put you in the position of a component inside it.
• Global and local views: Explain that an all-knowing observer can see the state of the entire system, whereas a component can observe only its own state. State the resulting challenge as thinking globally while acting locally.
  Why it matters: System-wide goals must be pursued by components that cannot directly see the whole system.

Key terms: black-box, white-box, resolution, perspective, global view, local view, thinking globally while acting locally

1.7.4 Think globally; act locally

This section frames distributed-system design as the challenge of achieving a global guarantee through components that can act only on local information.

• Global guarantees from local actions: Explain why a distributed system needs a global algorithm implemented through local algorithms, even though each component has access only to local information. Use the splitbrain example to describe the difficulty of ensuring that only one participant believes they are the leader.
  Why it matters: This distinction helps you reason about whether local decisions will produce the behavior the whole system requires.

Key terms: global algorithm, local algorithm, global correctness, local knowledge, splitbrain

1.8 Thinking above the code

This section uses a code-free model of concurrent processes to explain race conditions and introduce reasoning about serializability.

• Models above the code: Explain why the section favors tailored mental models over static code snippets for reasoning about dynamic systems, using the narrow impression a two-thread, single-variable example can give of race conditions.
  Why it matters: A useful model exposes the general problem rather than tying it to one code example.
• Interleavings and race conditions: Describe a process as an ordered sequence of atomic actions and concurrent composition as all possible interleavings of two processes. Explain that a race condition exists when some of those interleavings are judged incorrect, with correctness determined by the requirements at hand.
  Why it matters: This definition lets you recognize races across threads, machines, and other forms of concurrency.
• Sequential equivalence of executions: State the database example’s correctness criterion: every possible interleaving of two processes must have a result equivalent to running the first process entirely before the second or the second entirely before the first.
  Why it matters: It gives a concrete standard for judging concurrent outcomes without requiring the processes to run one after another.

Key terms: thinking above the code, tailored mental models, race conditions, process, atomic actions, concurrent composition, interleavings, race condition, correctness, sequential composition

1.9 Summary

The chapter summary connects mental models, the local-view challenge of distributed systems, and the guarantees used to reason about them.