Single Points of Failure & Resilience

A Single Point of Failure (SPOF) is any component whose failure takes down the entire system. If one box dying means your service dies, that box is an SPOF.

Where SPOFs Hide

• Centralized databases — one DB instance serves everything.
• Coordinators and proxies — components handling routing, service discovery, or job assignment are especially prone: everything flows through them by design.
• Single load balancer — ironic but common; the component meant to add resilience becomes the choke point.

The Removal Toolkit

Technique	What it does
Redundancy	Run multiple instances of every service; health-check them
Replication	Duplicate data across independent machines
Failover	Automatically switch to a standby on failure (e.g. promote a DB slave to master)
Horizontal scaling	More nodes = no single node is load-bearing
Partitioning	Spread load/data across systems so failures are isolated

Backups

Backups don’t prevent failure — they bound its damage:

• Keep regular backups of all stateful stores.
• A backup turns “total data loss” into “restore from last night”.
• Pair with redundancy so recovery is fast, not just possible.

The Dependency Graph

Map what depends on what. With enough redundancy the dependency graph becomes flexible — traffic reroutes around failed nodes instead of through them. Audit it regularly: new features quietly introduce new SPOFs.

Limits: CAP

You can’t remove every failure mode:

• CAP theorem constrains what’s achievable during partitions — you trade consistency against availability.
• Strong consistency often requires coordination (quorums, leader election), which itself can reduce availability.

Resilience engineering is about choosing which failures you can tolerate, then making those failures boring.

---

Part of the System Design Fundamentals series.