Communication is the exchange of data and control signals among distributed-system nodes. It supports replica synchronization, shard reassignment, and coordinated health checks, making reliable messaging central to operational reliability. section 6.6
Assess communication choices through their consistency, latency, and failure-handling trade-offs:
- Synchronous replication completes a write after all replicas acknowledge it, strengthening consistency while increasing write latency. section 6.6
- Asynchronous replication acknowledges after the primary processes the write, reducing latency while permitting temporary replica inconsistencies. section 6.6
- Apply RPC or queues for replication events, reassignments, and health checks; use retries, timeouts, and backpressure to manage unreliable communication. section 6.6
- Optimize communication for locality and tune batching or compression when transferring large migrations. section 6.6
Interview prompt
When asked to compare synchronous and asynchronous replication, explain first when a write is acknowledged, then connect that decision to consistency, latency, and fault tolerance. Expect follow-up questions about reliable node-to-node messaging, retries, timeouts, and backpressure. section 6.6
Table 6.7 Communication, observability, and monitoring
|
Topic
|
Fundamentals
|
Applied skills
|
Advanced
|
|
Communication
|
Understand reliable node-to-node messaging. Know synchronous vs. asynchronous replication.
|
Use RPC/queues for replication events, reassignments, health checks; apply retry, timeout, backpressure.
|
Optimize topologies for locality, tune batching/compression for large migrations.
|
|
Observability
|
Know that metrics, logs, and traces reveal system state. Understand key metrics: replication lag, shard balance, partition health.
|
Configure distributed tracing and structured logging to follow requests across replicas and partitions. Identify bottlenecks in cross-node queries.
|
Build unified observability pipelines for metrics/logs/traces. Enable anomaly detection and predictive analytics for shard rebalancing or replication lag.
|
|
Monitoring
|
Understand the need for continuous health checks and alerting. Recognize common signals: latency, error rates, throughput drops.
|
Set up dashboards and alerts for replication delays, shard hotspots, partition failures. Use SLAs/SLOs to guide monitoring thresholds.
|
Automate incident response—trigger self-healing, dynamic scaling, or traffic rerouting when anomalies are detected. Integrate with on-call workflows for rapid resolution.
|
Use the comparison table to scan communication fundamentals, applied mechanisms, and advanced optimization practices alongside observability and monitoring. section 6.6








