System Design Notes

Monolith vs Microservices

Sun, 23 Aug 2026 00:00:00 GMT

A monolith is a single application (running on one or many machines) where all computation lives in one deployable unit. A microservice architecture divides the whole workload into business units, each becoming an independent service with its own data and deployment.

Neither is "better" — they optimize for different failure modes of software teams.

Side-by-Side

Dimension	Monolith	Microservices
Moving parts	Few	Many
Fit	Small teams, early stage	Larger orgs, independent domains
Duplication (tests, tooling)	Less	More per service
Speed	Faster — in-process procedure calls	Network calls (RPC) per interaction
Deployment	Complex — everything touches everything	Independent per service
Blast radius	One bug can break everything	Isolated to one service
Coupling	Tight	Loose (by contract)
Scaling	Scale the whole thing	Scale exactly what's hot
Context needed to work	Whole codebase	One service at a time
Design effort	Lower upfront	Tougher — boundaries must be right

Monolith

Advantages: fewer moving parts; great for small teams; less duplicated test/tooling infrastructure; fast because everything runs in the same box.

Disadvantages: working on anything requires more context; deployments are complex and risky; too much responsibility concentrated in one server — one bad change can break everything; tight coupling makes evolution slow.

Microservices

Advantages: easy to scale individual services; less context needed per developer; parallel development across teams; resources can be allocated where load actually is.

Disadvantages: tougher to design — service boundaries, data ownership, inter-service contracts, and distributed-systems failure modes all become your problem.

How to Choose

Start monolith until domain boundaries are proven. Splitting later is far easier than un-splitting wrong boundaries.
Interview framing: most interview questions assume high scale, so microservices are the expected answer — but say why (independent scaling, team autonomy) and acknowledge the operational price.
The migration path is usually: monolith → extract the hot/critical path as a service → repeat. See Microservices Architecture Patterns for decomposition strategies once you do split.

Part of the Architecture Patterns series.

Database Sharding

Sun, 23 Aug 2026 00:00:00 GMT

Sharding is horizontal partitioning of a database into smaller, independent pieces (shards), each holding a disjoint subset of rows and running on its own hardware. You shard when a single database genuinely cannot handle the write volume or dataset size — not before.

The Shard Key

The shard key decides how data is split and where each row lives. Choosing it is the most consequential decision in sharding:

A good key spreads load evenly.
A bad key creates hot shards — a few shards take most of the traffic while others idle.

Example: sharding by user_id spreads users evenly; sharding by created_at puts all current traffic on one shard (everyone writes to "today").

Common Strategies

Strategy	How it works	Trade-off
Range-based	Shard 1: IDs 1–1M, Shard 2: 1M–2M…	Simple range queries; uneven load risk
Hash-based	`hash(key) % n` picks the shard	Even distribution; range scans hit all shards

For dynamic shard counts, consistent hashing avoids mass reassignment when shards are added (see Consistent Hashing).

Routing

A routing layer — application logic or a proxy — maps each request to the correct shard. This layer must know the partitioning scheme and adds one more component that can fail or become a bottleneck if not made highly available.

Replication Per Shard

Each shard is itself replicated for availability:

One master handles all writes; slaves serve reads.
If the master fails, a slave is promoted (failover).

This gives you both scalability (data spread across machines) and availability (no single machine's loss loses data).

Strengths

Improves scalability (write capacity grows with shards) and availability (blast radius shrinks per shard).
Shards can be split further only when they grow too large.

Costs & Complexity

Cross-shard queries are expensive — joins and transactions spanning shards need application-level coordination.
Rebalancing is difficult — moving data between shards while serving traffic requires careful tooling.
System complexity — routing, monitoring per shard, and debugging all get harder. Only pay this price when you truly need it.

Part of the Databases series.

Event-Driven Architecture

Sun, 23 Aug 2026 00:00:00 GMT

Event-driven systems communicate through events — records of state changes ("order placed", "payment failed") rather than direct commands.

Flow: Producer → Broker (queue/log) → Consumer(s)

Two properties make events powerful:

Immutable — never updated, only appended.
Persisted — stored durably, enabling replay and recovery.

Replayability

Because the event log is a complete history, system state can be rebuilt by reprocessing events from the beginning. This enables:

Self-healing — after a bug corrupts state, fix the handler and replay the log.
Debugging and auditing — every change has a traceable cause.
New consumers — subscribe and replay history to build a current view.

Brokers like Kafka and RabbitMQ provide the persistence, retry, and delivery machinery.

Reliability Requirements

Idempotency

At-least-once delivery means the same event can arrive multiple times. Handlers must be idempotent — processing an event twice must be safe. Common technique: track processed event IDs and dedupe.

Retries + ACKs

Consumers acknowledge messages only after successful processing. Unacked messages are redelivered — this is what makes delivery reliable, and also why idempotency is non-negotiable.

Event Sourcing & CQRS

Event Sourcing: store all changes as events, not final state. The database is the event log; current state is derived.
CQRS (Command Query Responsibility Segregation): separate the write model (commands producing events) from the read model (projections built from those events). Each side scales and evolves independently.

Strengths

Loose coupling — services evolve independently; add new consumers without touching producers.
Scalability — asynchronous processing absorbs spikes naturally.
Availability & fault tolerance — consumers can be down while events accumulate safely in the broker.

Weaknesses

No strict ordering guarantee by default — depends on partitioning/design.
Eventual consistency — reads may lag writes; wrong fit for strong transactional requirements.
Distributed debugging — tracing one request across many async consumers is hard without correlation IDs.
Careful handling required for retries, duplicates, ordering, and failure paths.

Real-World Uses

Payments, notifications, analytics pipelines
Git (the commit history is an event log)
Gaming (rebuilding game state from event history)

Not ideal when you need strong transactional consistency across a write — use a relational design there instead.

Part of the Messaging series.

Message Queues: Async Processing at Scale

Sun, 23 Aug 2026 00:00:00 GMT

With multiple servers processing jobs in parallel, a server can crash mid-job. The jobs it was holding are now unfinished — they must not be lost, and they must be reassigned.

The Reassignment Pattern

A monitor/notifier watches server health. On crash detection:

Fetch the crashed server's unfinished jobs from shared storage.
Redistribute them to alive servers.
Distribution uses consistent hashing via the load balancer.

Because consistent hashing maps the same job ID to the same server, jobs already running on healthy servers don't get duplicated — only the dead server's slice moves.

This pattern — durable job list, health checks, retry, redistribution — is essentially how a message queue works.

What a Message Queue Gives You

Producers enqueue work; consumers process it. Neither knows about the other directly.
Messages are persisted until acknowledged, so crashes don't lose work.
Consumers scale independently of producers (add workers at peak, drain backlog off-hours).
Failed messages can be retried or routed to a dead-letter queue.

The Pub-Sub Model

Pub-sub generalizes the queue from one consumer per message to many:

Publishers emit events ("order placed").
A broker (Kafka, RabbitMQ) routes them to topic subscribers.
Subscribers consume independently — notifications, analytics, fraud checks all react to the same event without touching each other.

Key properties:

Property	Effect
Services never call each other directly	Loose coupling; add consumers without touching producers
Asynchronous delivery	Producers aren't blocked by slow consumers
Broker owns persistence/retries	Reliable delivery even when consumers are down

The Trade-offs

Slower than direct calls — an extra network hop and serialization layer.
Eventual consistency — consumers lag producers; wrong fit when you need strong transactional guarantees.
Operational complexity — brokers need setup, monitoring, capacity planning.
Failure handling is your job — retries, duplicates, and ordering must be designed for (see Event-Driven Architecture for idempotency).

When to Reach For It

Work that doesn't need an immediate response (emails, thumbnails, analytics).
Spike absorption (enqueue now, process later).
Fan-out to multiple independent consumers.

Avoid it for request/response paths that need a synchronous answer.

Part of the Messaging series.

CDNs: Content Delivery Networks

Sun, 23 Aug 2026 00:00:00 GMT

A CDN is a network of servers distributed across the globe that serves content from the location nearest to each user. Physics is the win: shorter distance means lower latency, and no single origin carries the world's traffic.

What Belongs on a CDN

Primarily static content: images, CSS, JS bundles, videos, fonts — anything identical for every user. Dynamic/personalized responses need more care (and often shouldn't be cached at all).

Hit vs Miss Flow

Cache hit — an edge server near the user already holds the asset → served immediately.
Cache miss — edge fetches from the origin, stores a copy, serves it. Subsequent nearby users hit the now-warm cache.

This is caching with geographic distribution bolted on — all the usual concepts apply: TTLs, hit ratio as the key metric, and stale-data trade-offs.

Benefits

Benefit	Why
Lower latency	Content travels meters, not continents
Origin offloading	Most requests never reach your servers
Scalability & availability	Traffic spikes and even origin outages are absorbed by edges
Cost-effectiveness	Edge bandwidth is cheaper than scaling origin capacity

Invalidation

Cached content must sometimes change before its TTL expires (deploy day). CDNs support cache invalidation strategies — purging specific URLs or tags at the edges — plus pluggable fetch/caching behaviors to control what gets cached and revalidated.

Who Runs Them

CloudFront, Akamai, Cloudflare, Fastly — the pattern is commodity; you rarely build your own.

Part of the System Design Fundamentals series.

HLD vs LLD

Sun, 23 Aug 2026 00:00:00 GMT

System design interviews and real projects both operate at two altitudes.

HLD — High-Level Design

The big-picture architecture of a system:

Main modules and their responsibilities
Overall request/data flow
How components interact (sync vs async, which protocols)
Technology choices at the boundaries (DB type, queue, cache, CDN)

Deliverable: an architecture diagram you can defend — boxes, arrows, and the reasoning for each. Questions like "design Twitter" or "design a URL shortener" are HLD exercises. The scaling toolkit lives here: load balancers, sharding, caching, queues (see the other fundamentals notes).

LLD — Low-Level Design

The detailed design of each module from the HLD:

Classes, interfaces, and their relationships
Function signatures and API contracts
Database tables/schemas and indexes
Implementation logic and edge cases

Deliverable: code-shaped artifacts — class diagrams, schemas, API specs. Questions like "design a parking lot" or "design a BookMyShow seat-booking flow" are LLD exercises; they test object-oriented design and data modeling rather than scale.

How They Relate

	HLD	LLD
Question answered	What are the pieces and how do they talk?	How is each piece built inside?
Audience	Architects, senior engineers, interviewers	Implementing engineers
Output	Architecture diagram + component choices	Class/API/schema design
Failure mode if skipped	Building detailed the wrong thing	Vague hand-waving that collapses in code review

Rule of thumb: HLD first to make the big bets cheap to change, LLD second to make each bet buildable. A strong answer names the HLD components, then drills into LLD for the one or two most interesting components rather than shallowly covering everything.

Part of the System Design Fundamentals series.

Load Balancing & Consistent Hashing

Sun, 23 Aug 2026 00:00:00 GMT

A load balancer sits between clients and your server pool, distributing requests so no single machine is overwhelmed. It is mandatory equipment the moment you horizontally scale — multiple servers are useless if all traffic still lands on one of them.

The Naive Approach: Simple Hashing

The simplest distribution scheme is hashing:

server = hash(request_id) % n_servers

Same request ID always maps to the same server. Cheap, deterministic, and gives you cache/session locality for free.

The Drawback

n_servers changes whenever you add or remove a server — which is exactly what horizontal scaling is for. When n = 10 becomes n = 11, almost every key remaps to a different server:

Sessions stored in-memory are lost.
Caches go cold (most requests now miss).
Upstream connections churn.

At scale this disruption is unacceptable.

Consistent Hashing

Consistent hashing uses a fixed hash space (conceptually a ring, e.g. hash(key) % M where M is a huge constant like 2³²):

Servers are hashed onto the ring at specific positions.
Keys are hashed onto the same ring.
A key belongs to the next clockwise server.

Because the hash space never changes, adding or removing a server only reassigns the keys sitting between the old and new position — roughly 1/n of keys move instead of nearly all of them.

	Naive hashing	Consistent hashing
Keys moved when scaling	~all keys	~`1/n` of keys
Cache/session locality	Broken on every change	Mostly preserved
Complexity	Trivial	Ring + node lookup logic

Virtual Nodes

Physical servers don't sit evenly on the ring by chance — some get huge ranges, others tiny ones (hot spots).

Virtual nodes fix this: each physical server is placed on the ring many times (e.g. 100–200 replicas under different hash offsets). With enough virtual nodes, ranges even out statistically, and a failing server sheds its small slices to many neighbors instead of dumping one giant range onto one unlucky peer.

Where You'll See It

Distributed caches and stores: Redis Cluster, Cassandra, DynamoDB
Load balancers with sticky routing
Database sharding layers (see Sharding)

Interview Framing

When asked "how would you route jobs to workers?", walk through the progression: round-robin → hashing for stickiness → why % n breaks → hash ring → virtual nodes for balance. The story of why each step exists matters more than naming them.

Part of the System Design Fundamentals series.

Single Points of Failure & Resilience

Sun, 23 Aug 2026 00:00:00 GMT

A Single Point of Failure (SPOF) is any component whose failure takes down the entire system. If one box dying means your service dies, that box is an SPOF.

Where SPOFs Hide

Centralized databases — one DB instance serves everything.
Coordinators and proxies — components handling routing, service discovery, or job assignment are especially prone: everything flows through them by design.
Single load balancer — ironic but common; the component meant to add resilience becomes the choke point.

The Removal Toolkit

Technique	What it does
Redundancy	Run multiple instances of every service; health-check them
Replication	Duplicate data across independent machines
Failover	Automatically switch to a standby on failure (e.g. promote a DB slave to master)
Horizontal scaling	More nodes = no single node is load-bearing
Partitioning	Spread load/data across systems so failures are isolated

Backups

Backups don't prevent failure — they bound its damage:

Keep regular backups of all stateful stores.
A backup turns "total data loss" into "restore from last night".
Pair with redundancy so recovery is fast, not just possible.

The Dependency Graph

Map what depends on what. With enough redundancy the dependency graph becomes flexible — traffic reroutes around failed nodes instead of through them. Audit it regularly: new features quietly introduce new SPOFs.

Limits: CAP

You can't remove every failure mode:

CAP theorem constrains what's achievable during partitions — you trade consistency against availability.
Strong consistency often requires coordination (quorums, leader election), which itself can reduce availability.

Resilience engineering is about choosing which failures you can tolerate, then making those failures boring.

Part of the System Design Fundamentals series.

Microservices Architecture Patterns

Mon, 05 Feb 2024 00:00:00 GMT

Microservices architecture structures an application as a collection of loosely coupled, independently deployable services. This guide covers the essential patterns.

Decomposition Patterns

1. Decompose by Business Capability

Organize around business functions, not technical layers.

❌ Technical layers:
  ├─ UI Service
  ├─ Business Logic Service
  └─ Data Access Service

✅ Business capabilities:
  ├─ Order Management
  ├─ Customer Management
  ├─ Inventory Management
  ├─ Payment Processing
  └─ Shipping & Fulfillment

2. Decompose by Subdomain (DDD)

Use Domain-Driven Design to identify bounded contexts.

// Each bounded context = potential microservice
public class OrderContext {      // Order Management
    // Entities: Order, OrderItem, Shipment
    // Aggregates: Order (root)
}

public class CustomerContext {   // Customer Management
    // Entities: Customer, Address, PaymentMethod
    // Aggregates: Customer (root)
}

public class InventoryContext {  // Inventory Management
    // Entities: Product, Warehouse, StockLevel
    // Aggregates: Product (root)
}

3. Strangler Fig Pattern

Incrementally migrate from monolith.

Phase 1:                    Phase 2:                    Phase 3:
┌─────────────┐             ┌─────────────┐             ┌─────────────┐
│  Monolith   │             │  Monolith   │             │  Monolith   │
│             │             │  ┌───────┐  │             │  ┌───────┐  │
│  [Orders]   │──────────▶  │  [Orders]   │──────────▶  │  [Orders]   │
│  [Customer] │  Extract    │  [Customer] │  Extract    │  [Customer] │
│  [Inventory]│             │  [Inventory]│             │  [Inventory]│
└─────────────┘             │  ┌───────┐  │             │  ┌───────┐  │
                            │  │Order Svc│  │             │  │Order Svc│  │
                            │  └───────┘  │             │  └───────┘  │
                            └─────────────┘             └─────────────┘

Communication Patterns

1. Synchronous (Request-Response)

REST/HTTP

GET /api/v1/orders/{orderId}
Authorization: Bearer <token>
Accept: application/json

Response: 200 OK
{
  "orderId": "ord_123",
  "customerId": "cust_456",
  "status": "SHIPPED",
  "items": [...]
}

gRPC (Recommended for internal)

// order.proto
service OrderService {
  rpc GetOrder(GetOrderRequest) returns (Order);
  rpc CreateOrder(CreateOrderRequest) returns (Order);
  rpc StreamOrders(StreamOrdersRequest) returns (stream Order);
}

message Order {
  string order_id = 1;
  string customer_id = 2;
  OrderStatus status = 3;
  repeated OrderItem items = 4;
}

2. Asynchronous (Event-Driven)

// Event publication
public class OrderService {
    private final EventPublisher publisher;
    
    public Order createOrder(CreateOrderCommand cmd) {
        Order order = Order.create(cmd);
        orderRepository.save(order);
        
        // Publish domain event
        publisher.publish(new OrderCreatedEvent(
            order.getId(),
            order.getCustomerId(),
            order.getItems(),
            order.getTotal()
        ));
        
        return order;
    }
}

// Event consumption (in Payment Service)
@EventListener
public class PaymentEventHandler {
    private final PaymentService paymentService;
    
    @EventHandler
    public void handle(OrderCreatedEvent event) {
        paymentService.processPayment(
            event.getOrderId(),
            event.getCustomerId(),
            event.getTotal()
        );
    }
}

3. API Gateway Pattern

Single entry point for all clients.

# Kong API Gateway config
services:
  - name: order-service
    url: http://order-service:8080
    routes:
      - name: orders
        paths: ["/api/orders"]
        methods: ["GET", "POST"]
    plugins:
      - name: rate-limiting
        config:
          minute: 1000
          policy: redis
      - name: jwt
      - name: correlation-id

  - name: customer-service
    url: http://customer-service:8080
    routes:
      - name: customers
        paths: ["/api/customers"]

4. Backend for Frontend (BFF)

Tailored APIs per client type.

                    ┌─────────────┐
                    │   Web BFF   │──▶ Web App
                    └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
        ┌─────────┐  ┌─────────┐  ┌─────────┐
        │ Orders  │  │Customer │  │Inventory│
        │ Service │  │ Service │  │ Service │
        └─────────┘  └─────────┘  └─────────┘
              │            │            │
              └────────────┼────────────┘
                           ▼
                    ┌─────────────┐
                    │  Mobile BFF │──▶ Mobile App
                    └─────────────┘

Data Management Patterns

1. Database per Service

Each service owns its data.

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Order     │     │  Customer   │     │ Inventory   │
│  Service    │     │  Service    │     │  Service    │
│  ┌───────┐  │     │  ┌───────┐  │     │  ┌───────┐  │
│  │OrderDB│  │     │  │CustDB │  │     │  │InvDB  │  │
│  └───────┘  │     │  └───────┘  │     │  └───────┘  │
└─────────────┘     └─────────────┘     └─────────────┘

2. Saga Pattern (Distributed Transactions)

// Choreography-based saga
public class OrderSaga {
    private final EventPublisher publisher;
    
    public void createOrder(CreateOrderCommand cmd) {
        // Step 1: Create pending order
        Order order = Order.createPending(cmd);
        orderRepository.save(order);
        
        // Step 2: Reserve inventory (async)
        publisher.publish(new ReserveInventoryCommand(
            order.getId(), order.getItems()
        ));
    }
    
    @EventHandler
    public void handle(InventoryReservedEvent event) {
        Order order = orderRepository.find(event.getOrderId());
        order.confirmInventoryReserved();
        orderRepository.save(order);
        
        // Step 3: Process payment
        publisher.publish(new ProcessPaymentCommand(
            order.getId(), order.getCustomerId(), order.getTotal()
        ));
    }
    
    @EventHandler
    public void handle(PaymentProcessedEvent event) {
        Order order = orderRepository.find(event.getOrderId());
        order.confirmPayment();
        orderRepository.save(order);
        
        // Step 4: Create shipment
        publisher.publish(new CreateShipmentCommand(order));
    }
    
    // Compensating transactions
    @EventHandler
    public void handle(InventoryReservationFailedEvent event) {
        Order order = orderRepository.find(event.getOrderId());
        order.cancel("Inventory unavailable");
        orderRepository.save(order);
    }
}

3. Event Sourcing

Store state changes as events.

// Event-sourced aggregate
public class Order {
    private String orderId;
    private OrderStatus status;
    private List<OrderItem> items;
    private List<DomainEvent> uncommittedEvents = new ArrayList<>();
    
    public static Order create(CreateOrderCommand cmd) {
        Order order = new Order();
        order.apply(new OrderCreatedEvent(
            cmd.getOrderId(), cmd.getCustomerId(), cmd.getItems()
        ));
        return order;
    }
    
    public void addItem(AddItemCommand cmd) {
        if (status != OrderStatus.PENDING) {
            throw new IllegalStateException("Cannot modify confirmed order");
        }
        apply(new ItemAddedEvent(orderId, cmd.getItem()));
    }
    
    private void apply(DomainEvent event) {
        when(event);
        uncommittedEvents.add(event);
    }
    
    private void when(OrderCreatedEvent e) {
        this.orderId = e.getOrderId();
        this.status = OrderStatus.PENDING;
        this.items = new ArrayList<>();
    }
    
    private void when(ItemAddedEvent e) {
        this.items.add(e.getItem());
    }
}

4. CQRS (Command Query Responsibility Segregation)

Separate read and write models.

// Write model (commands)
public class OrderCommandHandler {
    private final EventStore eventStore;
    
    @CommandHandler
    public void handle(CreateOrderCommand cmd) {
        Order order = Order.create(cmd);
        eventStore.save(order.getUncommittedEvents());
    }
}

// Read model (queries) - optimized for reading
@Entity
@Table(name = "order_summary")
public class OrderSummary {
    @Id String orderId;
    String customerName;
    OrderStatus status;
    BigDecimal total;
    LocalDateTime createdAt;
    int itemCount;
}

// Projector updates read model
@EventHandler
public class OrderProjector {
    private final OrderSummaryRepository readRepo;
    
    public void on(OrderCreatedEvent event) {
        readRepo.save(new OrderSummary(
            event.getOrderId(),
            event.getCustomerName(),
            OrderStatus.PENDING,
            event.getTotal(),
            event.getTimestamp(),
            event.getItems().size()
        ));
    }
}

Operational Patterns

1. Service Discovery

# Consul service registration
service:
  name: order-service
  port: 8080
  tags: ["v1", "api"]
  check:
    http: http://localhost:8080/health
    interval: 10s
    timeout: 5s

2. Configuration Management

# Spring Cloud Config / Consul Config
order-service:
  database:
    url: ${DATABASE_URL}
    pool-size: 20
  inventory:
    service-url: http://inventory-service
    timeout: 5s
    retry-attempts: 3
  circuit-breaker:
    failure-threshold: 50%
    timeout: 30s

3. Distributed Tracing

// OpenTelemetry instrumentation
@SpanAttribute
public Order getOrder(String orderId) {
    return tracer.spanBuilder("getOrder")
        .setAttribute("order.id", orderId)
        .startScopedSpan(() -> {
            return orderRepository.findById(orderId);
        });
}

4. Health Checks

@Component
public class OrderHealthIndicator implements HealthIndicator {
    private final OrderRepository repository;
    private final InventoryClient inventoryClient;
    
    @Override
    public Health health() {
        // Check database
        if (!repository.isHealthy()) {
            return Health.down()
                .withDetail("database", "unavailable")
                .build();
        }
        
        // Check dependencies
        if (!inventoryClient.isHealthy()) {
            return Health.up()
                .withDetail("inventory", "degraded")
                .withDetail("database", "healthy")
                .build();
        }
        
        return Health.up()
            .withDetail("database", "healthy")
            .withDetail("inventory", "healthy")
            .build();
    }
}

Anti-Patterns to Avoid

Anti-Pattern	Problem	Solution
Distributed Monolith	Services must deploy together	Ensure independent deployability
Shared Database	Tight coupling, schema conflicts	Database per service
Chatty Communication	Latency, cascade failures	Async events, batch APIs
Synchronous Chains	A→B→C→D blocks	Event-driven, saga
Data Duplication	Inconsistency	Single source of truth + events
Nanoservices	Operational overhead	Right-size services

Technology Choices

Category	Options
Service Mesh	Istio, Linkerd, Consul Connect
API Gateway	Kong, AWS API Gateway, Traefik, Zuul
Service Discovery	Consul, etcd, Eureka, Kubernetes DNS
Message Broker	Kafka, RabbitMQ, NATS, Pulsar
Distributed Tracing	Jaeger, Zipkin, Tempo, AWS X-Ray
Metrics	Prometheus + Grafana, Datadog, CloudWatch
Logging	ELK/EFK, Loki, Splunk
Deployment	Kubernetes, Nomad, ECS, Cloud Run

Netflix Architecture: Microservices at Scale

Thu, 01 Feb 2024 00:00:00 GMT

Netflix is the canonical example of microservices architecture at massive scale. This case study explores their architecture, key decisions, and lessons learned.

Scale Numbers (2024)

230M+ subscribers across 190 countries
15,000+ titles in catalog
1M+ requests/second at peak
15% of global internet traffic (peak hours)
4,000+ microservices
1,000+ deployments/day

High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                        CDN (Open Connect)                    │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐        │
│  │  ISP 1  │  │  ISP 2  │  │  ISP 3  │  │  ...    │        │
│  └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘        │
└───────┼────────────┼────────────┼────────────┼──────────────┘
        │            │            │            │
        ▼            ▼            ▼            ▼
┌─────────────────────────────────────────────────────────────┐
│                      AWS Cloud                               │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐       │
│  │   API GW     │  │   Zuul GW    │  │   Edge       │       │
│  │   (GraphQL)  │  │   (Routing)  │  │   Services   │       │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘       │
└─────────┼─────────────────┼─────────────────┼────────────────┘
          │                 │                 │
    ┌─────▼─────┐    ┌──────▼──────┐    ┌─────▼─────┐
    │  User     │    │  Content    │    │  Playback │
    │  Service  │    │  Catalog    │    │  Service  │
    └─────┬─────┘    └──────┬──────┘    └─────┬─────┘
          │                 │                 │
    ┌─────▼─────────────────▼─────────────────▼─────┐
    │           Data Layer (Cassandra, Redis, S3)   │
    └────────────────────────────────────────────────┘

Key Architectural Decisions

1. Microservices with Bounded Contexts

Each service owns a single business capability:

# Service examples
services:
  user-service:
    owns: [profiles, preferences, accounts, billing]
    data: Cassandra (user profiles), PostgreSQL (billing)
  
  catalog-service:
    owns: [titles, metadata, search, recommendations]
    data: Elasticsearch, Cassandra
  
  playback-service:
    owns: [streaming, licensing, DRM, quality selection]
    data: Redis (session), S3 (manifests)
  
  discovery-service:
    owns: [search, browse, personalization]
    data: Elasticsearch, Cassandra

2. API Gateway Pattern

Zuul (Netflix's gateway) handles:

Request routing
Authentication/authorization
Rate limiting
Circuit breaking
Request/response transformation
A/B testing routing

// Zuul filter example
public class RateLimitFilter extends ZuulFilter {
    @Override
    public String filterType() { return "pre"; }
    
    @Override
    public int filterOrder() { return 1; }
    
    @Override
    public boolean shouldFilter() { return true; }
    
    @Override
    public Object run() {
        RequestContext ctx = RequestContext.getCurrentContext();
        String userId = ctx.getRequest().getHeader("X-User-Id");
        
        if (!rateLimiter.tryAcquire(userId)) {
            ctx.setSendZuulResponse(false);
            ctx.setResponseStatusCode(429);
            ctx.setResponseBody("Rate limit exceeded");
        }
        return null;
    }
}

3. Circuit Breaker Pattern (Hystrix)

Prevent cascade failures:

@HystrixCommand(
    fallbackMethod = "getFallbackCatalog",
    commandProperties = {
        @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "1000"),
        @HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "20"),
        @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "50"),
        @HystrixProperty(name = "circuitBreaker.sleepWindowInMilliseconds", value = "5000")
    }
)
public Catalog getCatalog(String userId) {
    return catalogClient.getRecommendations(userId);
}

public Catalog getFallbackCatalog(String userId) {
    // Return cached/popular content
    return catalogCache.getPopular();
}

4. Chaos Engineering (Simian Army)

Proactively inject failures:

Tool	Purpose
Chaos Monkey	Randomly terminates instances
Latency Monkey	Adds artificial latency
Conformity Monkey	Finds non-compliant instances
Doctor Monkey	Removes unhealthy instances
Janitor Monkey	Cleans up unused resources
Chaos Gorilla	Simulates AZ outage
Chaos Kong	Simulates region outage

5. Data Architecture

Cassandra for:

User viewing history (time-series)
Playback positions
Device registry
A/B test assignments

Redis for:

Session management
Rate limiting counters
Real-time recommendations cache
Feature flags

S3 for:

Video manifests (HLS/DASH)
Thumbnails, artwork
Backup/restore

Elasticsearch for:

Search indexing
Log aggregation
Metrics

Resilience Patterns

1. Bulkhead Pattern

Isolate failures to prevent cascade:

# Thread pool isolation per dependency
hystrix:
  threadpool:
    catalogService:
      coreSize: 20
      maxQueueSize: 100
    userService:
      coreSize: 10
      maxQueueSize: 50
    playbackService:
      coreSize: 50
      maxQueueSize: 200

2. Timeout and Retry Budgets

// Retry with exponential backoff + jitter
@Retryable(
    maxAttempts = 3,
    backoff = @Backoff(delay = 100, multiplier = 2, random = true),
    value = {TimeoutException.class, ConnectException.class}
)
public Catalog fetchCatalog(String userId) {
    return catalogClient.getCatalog(userId);
}

3. Graceful Degradation

public PlaybackManifest getManifest(String titleId, String deviceId) {
    try {
        return manifestService.getPersonalizedManifest(titleId, deviceId);
    } catch (Exception e) {
        log.warn("Personalized manifest failed, using generic", e);
        return manifestService.getGenericManifest(titleId);
    }
}

4. Request Collapsing

// Collapse duplicate concurrent requests
@RequestCollapser(
    scope = RequestCollapser.Scope.REQUEST,
    collapserKey = "getUserProfile"
)
public UserProfile getUserProfile(String userId) {
    return userClient.getProfile(userId);
}

Deployment Pipeline

┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐
│  Code   │──▶│  Build  │──▶│  Test   │──▶│  Canary │──▶│  Prod   │
│  Commit │   │  & Unit │   │  Int.   │   │  Deploy │   │  Rollout│
└─────────┘   └─────────┘   └─────────┘   └─────────┘   └─────────┘
                 │             │             │             │
          ┌──────▼──────┐ ┌───▼────┐ ┌──────▼──────┐ ┌────▼────┐
          │  Spinnaker  │ │ Bakery │ │  Kayenta    │ │  Atlas  │
          │  Pipeline   │ │  (AMI) │ │  (Canary    │ │  (Metrics)│
          └─────────────┘ └────────┘ │   Analysis) │ └─────────┘
                                    └─────────────┘

Spinnaker Pipeline Stages:

Bake: Create immutable AMI
Deploy to Dev: Automated
Integration Tests: Automated
Canary Deploy: 1% → 10% → 100%
Kayenta Analysis: Automated metric comparison
Production Rollout: Gradual with pause points

Key Lessons Learned

Invest in observability early: Metrics, logs, traces, dashboards
Automate everything: Deployments, rollbacks, capacity, failover
Design for failure: Assume everything will fail
Culture of ownership: You build it, you run it
Incremental migration: Strangler fig pattern from monolith
Standardize but allow innovation: Paved road + experimentation
Chaos engineering is not optional: Test failure scenarios regularly

Modern Evolution (2020+)

GraphQL Federation: Unified API layer
gRPC: Service-to-service communication
Kubernetes: Container orchestration (migrating from Titus)
Serverless: AWS Lambda for sporadic workloads
ML Platform: Metaflow for ML workflows

Caching Strategies: Patterns and Best Practices

Thu, 25 Jan 2024 00:00:00 GMT

Caching is one of the highest-leverage optimizations in system design. A well-designed cache can reduce latency by 10-100x and reduce database load by 90%+.

Why Cache?

Latency: Memory (ns-μs) vs Disk (ms) vs Network (ms)
Throughput: Reduce load on primary data store
Cost: Serve more requests with same infrastructure
Availability: Cache can serve stale data during outages

Cache Patterns

1. Cache-Aside (Lazy Loading)

Most common pattern. Application manages cache.

Read:
  1. Check cache
  2. If hit → return data
  3. If miss → query DB → store in cache → return data

Write:
  1. Update DB
  2. Invalidate/Update cache

# Cache-aside implementation
class CacheAside:
    def __init__(self, cache, db):
        self.cache = cache
        self.db = db
    
    def get(self, key: str):
        # Try cache first
        value = self.cache.get(key)
        if value is not None:
            return value
        
        # Cache miss - fetch from DB
        value = self.db.get(key)
        if value is not None:
            self.cache.set(key, value, ttl=300)
        return value
    
    def set(self, key: str, value: any):
        # Write to DB first
        self.db.set(key, value)
        # Then invalidate cache (or update)
        self.cache.delete(key)

Pros: Simple, cache only stores hot data, resilient to cache failures Cons: Cache miss penalty, potential for stale data on write

2. Read-Through

Cache sits between app and DB. Cache handles misses.

Read:
  1. App requests from cache
  2. Cache checks → if miss, loads from DB → returns to app

# Read-through cache (transparent to app)
class ReadThroughCache:
    def __init__(self, cache, db):
        self.cache = cache
        self.db = db
    
    def get(self, key: str):
        value = self.cache.get(key)
        if value is None:
            # Cache loads from DB automatically
            value = self.db.get(key)
            if value is not None:
                self.cache.set(key, value, ttl=300)
        return value

Pros: Application logic simpler, cache manages population Cons: More complex cache implementation, first request always slow

3. Write-Through

Writes go to cache and DB synchronously.

Write:
  1. App writes to cache
  2. Cache writes to DB (synchronously)
  3. Return success

class WriteThroughCache:
    def set(self, key: str, value: any):
        # Write to both cache and DB
        self.cache.set(key, value, ttl=300)
        self.db.set(key, value)  # Synchronous

Pros: Cache always consistent with DB, no stale reads Cons: Higher write latency, DB must be available

4. Write-Back (Write-Behind)

Writes go to cache first, DB updated asynchronously.

Write:
  1. App writes to cache
  2. Return success immediately
  2. Cache asynchronously writes to DB (batched)

import asyncio
from collections import deque

class WriteBackCache:
    def __init__(self, cache, db, flush_interval=5):
        self.cache = cache
        self.db = db
        self.write_queue = deque()
        self.flush_interval = flush_interval
        asyncio.create_task(self._flush_loop())
    
    def set(self, key: str, value: any):
        self.cache.set(key, value, ttl=300)
        self.write_queue.append((key, value))
    
    async def _flush_loop(self):
        while True:
            await asyncio.sleep(self.flush_interval)
            await self._flush()
    
    async def _flush(self):
        if not self.write_queue:
            return
        # Batch write to DB
        batch = []
        while self.write_queue:
            batch.append(self.write_queue.popleft())
        await self.db.batch_set(batch)

Pros: Lowest write latency, batches DB writes Cons: Data loss risk on cache failure, complex implementation

5. Refresh-Ahead

Proactively refresh cache before expiration.

class RefreshAheadCache:
    def __init__(self, cache, db, refresh_threshold=0.8):
        self.cache = cache
        self.db = db
        self.refresh_threshold = refresh_threshold  # 80% of TTL
    
    async def get(self, key: str):
        value, ttl = await self.cache.get_with_ttl(key)
        
        # Proactively refresh if near expiration
        if value is not None and ttl < self.cache.default_ttl * self.refresh_threshold:
            asyncio.create_task(self._refresh(key))
        
        if value is None:
            value = await self.db.get(key)
            if value:
                await self.cache.set(key, value)
        return value
    
    async def _refresh(self, key: str):
        value = await self.db.get(key)
        if value:
            await self.cache.set(key, value)

Cache Invalidation Strategies

Time-Based (TTL)

# Simple TTL
cache.set(key, value, ttl=300)  # 5 minutes

Event-Based (Explicit Invalidation)

# On write, invalidate related keys
def update_user(user_id, data):
    db.update_user(user_id, data)
    cache.delete(f"user:{user_id}")
    cache.delete(f"user:{user_id}:profile")
    cache.delete(f"user:{user_id}:posts")

Key-Based Invalidation (Versioning)

# Include version in key
cache_key = f"user:{user_id}:v{user.version}"

# On update, increment version
user.version += 1
db.save(user)
# Old cache key becomes unreachable

Cache Tags (Redis 7+)

# Tag related keys
SET user:1:profile "..." EX 300
SADD tag:user:1 user:1:profile

# Invalidate all user:1 keys
SMEMBERS tag:user:1 → DEL each key

Common Pitfalls

1. Thundering Herd

Multiple requests hit expired cache simultaneously.

# Solution: Use distributed lock
async def get_with_lock(key: str):
    value = await cache.get(key)
    if value:
        return value
    
    # Try to acquire lock
    lock = await redis.set(f"lock:{key}", "1", nx=True, ex=10)
    if lock:
        try:
            value = await db.get(key)
            await cache.set(key, value)
            return value
        finally:
            await redis.delete(f"lock:{key}")
    else:
        # Wait for other request to populate
        await asyncio.sleep(0.1)
        return await cache.get(key)

2. Cache Stampede (Dogpile Effect)

Same as thundering herd - use probabilistic early expiration.

# Add jitter to TTL
import random
ttl = base_ttl + random.randint(0, base_ttl // 4)
cache.set(key, value, ttl=ttl)

3. Hot Keys

Single key gets disproportionate traffic.

# Solution: Key sharding + local cache
def get_sharded(key: str):
    # Split hot key across multiple cache entries
    shard = hash(key) % 10
    return cache.get(f"{key}:shard:{shard}")

4. Cache Pollution

Cache filled with rarely-accessed data.

# Solution: LRU eviction + frequency-based admission
# Redis: maxmemory-policy allkeys-lfu
# Or use TinyLFU / W-TinyLFU (Ristretto, Caffeine)

Multi-Level Caching

Request → CDN (edge) → API Gateway → Redis (distributed) → Local (in-memory) → DB
              ↓              ↓               ↓                ↓
           Static         Auth/Rate       Session/         Hot data
           Assets         Limit           User Data

// Multi-level cache in Go
type MultiLevelCache struct {
    local  *ristretto.Cache  // In-process, ~100MB
    shared *redis.Client     // Distributed
}

func (m *MultiLevelCache) Get(key string) (string, bool) {
    // L1: Local cache (fastest)
    if val, ok := m.local.Get(key); ok {
        return val.(string), true
    }
    
    // L2: Shared cache
    val, err := m.shared.Get(ctx, key).Result()
    if err == nil {
        m.local.Set(key, val, 1)
        return val, true
    }
    
    return "", false
}

Cache Sizing

Cache Size	Hit Rate (Typical)	Use Case
1% of data	50-70%	Very hot subset
5% of data	80-90%	Most applications
10% of data	90-95%	Read-heavy workloads
20%+ of data	95%+	Specialized

Monitoring Metrics

# Key cache metrics
cache_hit_rate{instance="..."}  # Target: >90%
cache_miss_rate{instance="..."}
cache_eviction_rate{instance="..."}
cache_memory_usage_bytes{instance="..."}
cache_latency_p99{instance="..."}  # Target: <1ms

SQL vs NoSQL: Choosing the Right Database

Mon, 22 Jan 2024 00:00:00 GMT

Choosing between SQL and NoSQL is one of the most fundamental architectural decisions. This guide helps you make an informed choice based on your specific requirements.

Quick Comparison

Aspect	SQL (Relational)	NoSQL (Non-Relational)
Data Model	Tables, rows, columns	Documents, key-value, wide-column, graph
Schema	Fixed, enforced	Flexible, schema-on-read
Query Language	SQL (standardized)	Varies by database
Transactions	ACID (strong)	BASE (eventual)
Scaling	Vertical (primarily)	Horizontal (native)
Joins	Native support	Limited/denormalized
Consistency	Strong	Eventual (tunable)

SQL Databases

When to Choose SQL

Complex relationships: Foreign keys, joins, many-to-many
ACID transactions required: Financial, inventory, booking systems
Structured, stable schema: Well-defined entities that don't change often
Ad-hoc queries: Need flexible querying without knowing access patterns upfront
Mature ecosystem: Tooling, ORMs, migration tools, expertise

Popular SQL Databases

Database	Best For	Notable Features
PostgreSQL	General purpose, complex queries	JSONB, extensions, window functions
MySQL	Web applications, read-heavy	Simple, fast, wide adoption
SQL Server	Enterprise, .NET ecosystem	T-SQL, integration services
CockroachDB	Distributed SQL	Horizontal scaling, strong consistency
TiDB	HTAP (hybrid transactional/analytical)	MySQL compatible, distributed

Schema Design Example

-- Normalized schema for e-commerce
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE orders (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id),
    status VARCHAR(50) NOT NULL DEFAULT 'pending',
    total_cents INTEGER NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE order_items (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    order_id UUID REFERENCES orders(id),
    product_id UUID NOT NULL,
    quantity INTEGER NOT NULL,
    price_cents INTEGER NOT NULL
);

-- Indexes for common queries
CREATE INDEX idx_orders_user_id ON orders(user_id);
CREATE INDEX idx_orders_status_created ON orders(status, created_at DESC);

NoSQL Databases

Types of NoSQL

Document Databases (MongoDB, Couchbase)

Store JSON-like documents
Flexible schema, nested data
Good for: Content management, catalogs, user profiles

Key-Value Stores (Redis, DynamoDB, Riak)

Simple key → value mapping
Extremely fast reads/writes
Good for: Caching, sessions, shopping carts

Wide-Column Stores (Cassandra, HBase, ScyllaDB)

Column families, sparse columns
Time-series, high write throughput
Good for: Logging, metrics, sensor data

Graph Databases (Neo4j, Amazon Neptune, ArangoDB)

Nodes and relationships
Traversals, recommendations
Good for: Social networks, fraud detection, knowledge graphs

When to Choose NoSQL

Massive scale: Petabytes, millions of writes/second
Flexible schema: Rapidly changing data structures
Specific access patterns: Known query patterns, no ad-hoc queries
High availability: AP systems for global applications
Specialized workloads: Time-series, graphs, key-value

NoSQL Data Modeling

// MongoDB: Embedded vs Referenced
// Embedded (one-to-few, read together)
{
  _id: ObjectId("..."),
  title: "System Design Notes",
  author: { name: "John", email: "john@example.com" },
  sections: [
    { title: "CAP Theorem", content: "...", order: 1 },
    { title: "Consensus", content: "...", order: 2 }
  ],
  tags: ["distributed-systems", "architecture"],
  createdAt: ISODate("2024-01-15")
}

// Referenced (one-to-many, independent lifecycle)
{
  _id: ObjectId("..."),
  title: "System Design Notes",
  authorId: ObjectId("..."),  // Reference to users collection
  sectionIds: [ObjectId("..."), ObjectId("...")],  // Array of refs
  tags: ["distributed-systems", "architecture"]
}

Decision Framework

START: What's your data like?
├── Highly relational, complex joins?
│   └── YES → SQL (PostgreSQL)
├── Need ACID transactions?
│   └── YES → SQL (PostgreSQL, CockroachDB)
├── Massive scale, simple access patterns?
│   └── YES → NoSQL (Cassandra, DynamoDB)
├── Flexible schema, rapid iteration?
│   └── YES → NoSQL (MongoDB)
├── Graph relationships, traversals?
│   └── YES → Graph DB (Neo4j)
├── Key-value, caching, sessions?
│   └── YES → Redis, DynamoDB
└── Time-series, metrics, logs?
    └── YES → Wide-column (TimescaleDB, InfluxDB)

Hybrid Approaches

Many modern systems use polyglot persistence:

┌─────────────────────────────────────────────────┐
│                  Application                     │
└──────────┬──────────────┬───────────┬───────────┘
           │              │           │
    ┌──────▼──────┐ ┌─────▼────┐ ┌────▼────┐
    │ PostgreSQL  │ │  Redis   │ │Elastic  │
    │ (Orders,    │ │ (Cache,  │ │Search   │
    │  Users,     │ │  Sessions)│ │(Logs,   │
    │  Payments)  │ │          │ │  Docs)  │
    └─────────────┘ └──────────┘ └─────────┘

NewSQL: Best of Both Worlds

NewSQL databases provide horizontal scaling with ACID:

CockroachDB: PostgreSQL-compatible, geo-distributed
TiDB: MySQL-compatible, HTAP
Google Spanner: Global consistency with TrueTime
YugabyteDB: PostgreSQL-compatible, multi-cloud

-- CockroachDB: Globally distributed, strong consistency
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email STRING UNIQUE NOT NULL,
    region STRING NOT NULL
) 
WITH (replication_zone = 'global');

Checklist for Database Selection

[ ] What are the access patterns? (Known vs ad-hoc)
[ ] What are the consistency requirements? (Strong vs eventual)
[ ] What is the scale? (Current and projected)
[ ] What is the team expertise? (SQL vs NoSQL experience)
[ ] What are the operational constraints? (Managed vs self-hosted)
[ ] What are the compliance requirements? (ACID, audit trails)
[ ] What is the data lifecycle? (Retention, archival, GDPR)

Scalability Patterns: Vertical vs Horizontal

Sat, 20 Jan 2024 00:00:00 GMT

Scalability is a system's ability to handle increased load by adding resources. There are two fundamental approaches: vertical scaling (scale up) and horizontal scaling (scale out).

Vertical Scaling (Scale Up)

Adding more resources (CPU, RAM, storage) to a single machine.

Pros

Simple: No application changes needed
Strong consistency: Single database, no distributed transactions
Lower operational complexity: One server to manage

Cons

Hardware limits: Maximum CPU, RAM, storage per machine
Single point of failure: If the machine dies, everything dies
Cost: High-end hardware is disproportionately expensive
Downtime: Often requires restart for upgrades

When to Use

Early-stage startups with predictable growth
Legacy applications hard to distribute
Workloads that don't parallelize well (some ML training)

# Example: AWS RDS vertical scaling
# r5.large → r5.xlarge → r5.2xlarge → r5.4xlarge
# vCPUs: 2 → 4 → 8 → 16
# Memory: 16GB → 32GB → 64GB → 128GB
# Cost/month: ~$180 → $360 → $720 → $1440

Horizontal Scaling (Scale Out)

Adding more machines to distribute the load.

Pros

No practical limits: Can scale to thousands of machines
Fault tolerance: Failure of one node doesn't bring down system
Cost-effective: Commodity hardware, linear cost scaling
Elasticity: Add/remove nodes based on demand

Cons

Complexity: Distributed systems challenges (CAP, consensus, consistency)
Application changes: Code must be designed for distribution
Operational overhead: More servers to monitor, deploy, debug
Network latency: Inter-node communication adds latency

When to Use

High-growth applications
Systems requiring high availability
Microservices architectures
Big data processing

Scaling Patterns

1. Load Balancing

Distribute incoming requests across multiple servers.

# NGINX load balancer config
upstream backend {
    least_conn;  # Least connections algorithm
    server 10.0.0.1:8080 weight=3;
    server 10.0.0.2:8080 weight=2;
    server 10.0.0.3:8080 weight=1;
    
    # Health checks
    server 10.0.0.4:8080 backup;
}

server {
    location / {
        proxy_pass http://backend;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

2. Stateless Services

Services that don't store session state locally.

Client → Load Balancer → [Service A, Service B, Service C]
                              ↓
                        Shared Cache (Redis)
                              ↓
                        Database

3. Database Sharding

Partition data across multiple database instances.

# Simple hash-based sharding
def get_shard(user_id: int, num_shards: int) -> int:
    return hash(user_id) % num_shards

# Consistent hashing (better for dynamic scaling)
import hashlib

class ConsistentHashRing:
    def __init__(self, nodes: list, replicas: int = 150):
        self.ring = {}
        self.sorted_keys = []
        
        for node in nodes:
            for i in range(replicas):
                key = self._hash(f"{node}:{i}")
                self.ring[key] = node
                self.sorted_keys.append(key)
        self.sorted_keys.sort()
    
    def get_node(self, key: str) -> str:
        hash_key = self._hash(key)
        for node_hash in self.sorted_keys:
            if node_hash >= hash_key:
                return self.ring[node_hash]
        return self.ring[self.sorted_keys[0]]

4. Read Replicas

Scale reads by replicating data.

-- Primary handles writes
INSERT INTO orders (...) VALUES (...);

-- Replicas handle reads
SELECT * FROM orders WHERE user_id = ?;  -- Routed to replica

5. Caching Layers

Reduce database load with strategic caching.

Request → CDN → API Gateway → Redis Cache → Database
              ↓              ↓
           Static          Computed
           Assets          Responses

The Scalability Checklist

Layer	Vertical Approach	Horizontal Approach
Compute	Bigger VMs	More containers/pods
Database	Bigger instance	Sharding + read replicas
Cache	More memory	Cluster mode (Redis Cluster)
Storage	Bigger disks	Distributed FS (S3, HDFS)
Network	Higher bandwidth	Load balancers, CDN

Cost Comparison

Approach	Monthly Cost (Example)	Max Throughput
Vertical (r5.4xlarge)	~$1,440	~50K req/s
Horizontal (10 × r5.large)	~$1,800	~200K req/s
Horizontal (auto-scaled)	Variable	Unlimited

Anti-Patterns to Avoid

Distributed monolith: Services that must be deployed together
Chatty services: Excessive inter-service communication
Shared database: Multiple services sharing one database
Synchronous chains: A→B→C→D where each call blocks

Consensus Algorithms: Raft vs Paxos

Thu, 18 Jan 2024 00:00:00 GMT

Consensus is the problem of getting a group of nodes to agree on a single value. It's fundamental to building reliable distributed systems.

Why Consensus Matters

Leader election: Who coordinates?
State machine replication: Keeping replicas in sync
Configuration management: Cluster membership changes
Distributed locks: Coordinating access to resources

Paxos: The Classic Algorithm

Paxos was introduced by Leslie Lamport in 1998. It's notoriously difficult to understand and implement correctly.

Basic Paxos (Single-Decree)

Prepare Phase: Proposer sends prepare(n) to majority
Promise Phase: Acceptors promise not to accept proposals < n
Accept Phase: Proposer sends accept(n, value) to majority
Learn Phase: Learners learn the accepted value

Multi-Paxos

Optimizes for repeated consensus by electing a stable leader.

// Simplified Paxos proposer
type Proposer struct {
    proposalNumber int
    acceptors      []Acceptor
}

func (p *Proposer) Propose(value interface{}) error {
    // Phase 1: Prepare
    n := p.generateProposalNumber()
    promises := p.sendPrepare(n)
    
    if len(promises) < p.quorumSize() {
        return ErrNoQuorum
    }
    
    // Phase 2: Accept
    chosenValue := p.chooseValue(promises, value)
    accepted := p.sendAccept(n, chosenValue)
    
    if len(accepted) < p.quorumSize() {
        return ErrNoQuorum
    }
    
    return nil
}

Raft: Understandable Consensus

Raft was designed by Diego Ongaro and John Ousterhout in 2014 with the explicit goal of being understandable.

Key Differences from Paxos

Aspect	Paxos	Raft
Leader	Implicit, multiple possible	Explicit, single leader
Log Structure	Single-decree, complex composition	Continuous log entries
Membership Changes	Complex	Joint consensus (simple)
Readability	Low	High

Raft States

Follower → Candidate → Leader
    ↑_______________|

Follower: Passive, responds to RPCs
Candidate: Runs for leader, requests votes
Leader: Handles all client requests, replicates log

Raft Guarantees

Election Safety: At most one leader per term
Leader Completeness: Leader has all committed entries
State Machine Safety: Identical logs → identical state

Leader Election

// Raft election timeout (randomized 150-300ms)
func (rf *Raft) startElection() {
    rf.currentTerm++
    rf.votedFor = rf.me
    rf.state = Candidate
    
    votes := 1 // Vote for self
    for i := range rf.peers {
        if i == rf.me { continue }
        
        go func(server int) {
            args := RequestVoteArgs{
                Term:         rf.currentTerm,
                CandidateId:  rf.me,
                LastLogIndex: rf.getLastLogIndex(),
                LastLogTerm:  rf.getLastLogTerm(),
            }
            reply := RequestVoteReply{}
            if rf.sendRequestVote(server, &args, &reply) {
                rf.mu.Lock()
                if reply.VoteGranted {
                    votes++
                    if votes > len(rf.peers)/2 {
                        rf.becomeLeader()
                    }
                } else if reply.Term > rf.currentTerm {
                    rf.becomeFollower(reply.Term)
                }
                rf.mu.Unlock()
            }
        }(i)
    }
}

When to Use Which?

Use Raft When:

Building a new system (easier to implement correctly)
Team needs to understand and debug the consensus logic
Need clear membership change semantics
Using etcd, Consul, or TiKV

Use Paxos When:

Working with existing Paxos-based systems (Chubby, Spanner)
Need highly optimized, battle-tested implementations
Academic/research context

Production Systems Using Raft

etcd: Kubernetes configuration store
Consul: Service mesh configuration
TiKV: Distributed transactional KV store
CockroachDB: Distributed SQL (Raft variant)
Redis Cluster: Redis 7+ uses Raft for replication

CAP Theorem: Understanding the Trade-offs

Mon, 15 Jan 2024 00:00:00 GMT

The CAP theorem, formulated by Eric Brewer in 2000, states that a distributed data store can only simultaneously provide two out of three guarantees: Consistency, Availability, and Partition Tolerance.

The Three Properties

Consistency (C)

Every read receives the most recent write or an error. All nodes see the same data at the same time. This is linearizability - not to be confused with ACID consistency.

Availability (A)

Every request receives a (non-error) response, without guarantee that it contains the most recent write. The system remains operational even if some nodes fail.

Partition Tolerance (P)

The system continues to operate despite an arbitrary number of messages being dropped or delayed by the network between nodes. Network partitions are inevitable in distributed systems.

The Trade-off Space

Since network partitions will happen, you must choose between CP and AP:

System Type	Characteristics	Examples
CP (Consistency + Partition Tolerance)	Sacrifices availability during partitions. Returns errors or times out if it can't guarantee consistency.	MongoDB, Redis, HBase, ZooKeeper
AP (Availability + Partition Tolerance)	Sacrifices consistency during partitions. Always returns a response, but data may be stale.	Cassandra, DynamoDB, CouchDB, Riak
CA (Consistency + Availability)	Only possible in a single-node system. Not a practical choice for distributed systems.	Traditional RDBMS (single node)

PACELC Theorem

The CAP theorem only describes behavior during a partition. The PACELC theorem extends this:

If Partition (P) then Availability vs Consistency (A vs C); Else (E) Latency vs Consistency (L vs C)

Even without partitions, there's a trade-off between latency and consistency. Strong consistency requires coordination, which adds latency.

# Example: Choosing consistency level in Cassandra
from cassandra.cluster import Cluster
from cassandra import ConsistencyLevel

cluster = Cluster()
session = cluster.connect()

# Strong consistency - wait for quorum
session.execute(
    "SELECT * FROM users WHERE id = ?",
    [user_id],
    consistency_level=ConsistencyLevel.QUORUM
)

# Eventual consistency - faster reads
session.execute(
    "SELECT * FROM users WHERE id = ?",
    [user_id],
    consistency_level=ConsistencyLevel.ONE
)

Practical Decision Framework

Financial transactions, inventory → CP (Consistency critical)
Social media feeds, recommendations → AP (Availability critical)
User profiles, sessions → Often AP with read-repair
Configuration, metadata → CP (Strong consistency needed)

Modern Perspectives

Google Spanner: Claims to be CA using TrueTime (globally synchronized clocks)
Cosmos DB: Offers 5 consistency levels from Strong to Eventual
DynamoDB: Default eventual, supports strong consistent reads

The key insight: CAP is not a binary choice. Modern systems offer tunable consistency per operation.

System Design Notes

Monolith vs Microservices

Side-by-Side

Monolith

Microservices

How to Choose

Database Sharding

The Shard Key

Common Strategies

Routing

Replication Per Shard

Strengths

Costs & Complexity

Event-Driven Architecture

Replayability

Reliability Requirements

Idempotency

Retries + ACKs

Event Sourcing & CQRS

Strengths

Weaknesses

Real-World Uses

Message Queues: Async Processing at Scale

The Reassignment Pattern

What a Message Queue Gives You

The Pub-Sub Model

The Trade-offs

When to Reach For It

CDNs: Content Delivery Networks

What Belongs on a CDN

Hit vs Miss Flow

Benefits

Invalidation

Who Runs Them

HLD vs LLD

HLD — High-Level Design

LLD — Low-Level Design

How They Relate

Load Balancing & Consistent Hashing

The Naive Approach: Simple Hashing

The Drawback

Consistent Hashing

Virtual Nodes

Where You'll See It

Interview Framing

Single Points of Failure & Resilience

Where SPOFs Hide

The Removal Toolkit

Backups

The Dependency Graph

Limits: CAP

Microservices Architecture Patterns

Decomposition Patterns

1. Decompose by Business Capability

2. Decompose by Subdomain (DDD)

3. Strangler Fig Pattern

Communication Patterns

1. Synchronous (Request-Response)

REST/HTTP

gRPC (Recommended for internal)

2. Asynchronous (Event-Driven)

3. API Gateway Pattern

4. Backend for Frontend (BFF)

Data Management Patterns

1. Database per Service

2. Saga Pattern (Distributed Transactions)

3. Event Sourcing

4. CQRS (Command Query Responsibility Segregation)

Operational Patterns

1. Service Discovery

2. Configuration Management

3. Distributed Tracing

4. Health Checks

Anti-Patterns to Avoid

Technology Choices

Further Reading

Netflix Architecture: Microservices at Scale

Scale Numbers (2024)

High-Level Architecture

Key Architectural Decisions