Distributed Systems Fundamentals

Distributed systems fundamentals answer: why is building distributed systems hard? The mental model: (1) Networks fail silently. (2) Clocks are unreliable. (3) Processes can fail at any moment. These three facts force all the complexity of CAP, consensus, and consistency models. The CAP Theorem — Why You Can Never Have Everything In 2000, Eric Brewer conjectured, and in 2002 Gilbert & Lynch formally proved, that any distributed data store can guarantee at most two of three properties simultaneously: C — Consistency: every read sees the most recent write (linearizability) A — Availability: every request receives a non-error response P — Partition Tolerance: the system continues operating when network messages are lost or delayed Because network partitions are a physical reality (cables break, routers drop packets, AWS availability zones lose connectivity), P is not optional. The real choice is CP vs AP. CP Systems (prefer consistency over availability) When a partition occurs the system refuses requests rather than risk returning stale data. Why CP Master-controlled regions; refuses writes during split Zookeeper Leader steps down if it cannot reach majority System Cassandra (default) Multi-master replication DynamoDB (eventually consistent reads) Visualising the Trade-off PACELC — The Extension That Matters More in Practice CAP only discusses behaviour during a partition. Daniel Abadi's PACELC model (2012) adds what happens in the normal case (no partition): If P (partition): choose between A (availability) and C (consistency). E (else, no partition): choose between L (latency) and C (consistency). Real numbers matter here. Synchronous replication adds 5–50 ms per replica hop. For a global system with replicas in US-East and EU-West, that is +85 ms for every write if you want strong consistency. P trade-off PA PC PA PC Interview Q&A Q: Is a single-node database CA? A: Technically yes, but this is misleading. CA systems exist only in theory because any real distributed system faces partitions. Single-node systems sidestep CAP by not being distributed. Q: Can you tune CAP at read/write time? A: Yes — Cassandra's vs per statement lets you choose consistency per operation without restarting the cluster. Q: What does "strong consistency" mean for an interviewer? A: Linearizability — reads always reflect the latest committed write, as if there is only one copy of the data. Real-World CAP Decisions at Scale AWS DynamoDB — A Case Study in AP Design DynamoDB defaults to AP: reads from any replica without waiting for replication, meaning you may read stale data written milliseconds ago. The parameter switches to CP for that request, routing to the primary at a cost of 2x read capacity units and ~5 ms extra latency. DynamoDB Global Tables extend this to multi-region: each region is a master, writes replicate asynchronously within ~1 second globally. The trade-off is explicit: concurrent writes to the same item in different regions use Last-Write-Wins resolution, so two updates to the same field within 1 second may result in silent data loss.