System Design Concepts

1Client-server & APIsBasics

Clients send requests over the network to servers that own data and logic. The API is the contract between them: REST over HTTP is the default, gRPC for fast service-to-service calls, GraphQL when clients need flexible reads.

• Use nouns + HTTP verbs: GET /users/42, POST /orders
• Version APIs (/v1/) and paginate list endpoints
• Status codes matter: 2xx ok, 4xx caller error, 5xx server error

curl -X POST https://api.example.com/v1/orders \
  -H "Content-Type: application/json" \
  -d '{"itemId": 17, "qty": 2}'
# -> 201 Created  {"orderId": "ord_9f2", "status": "PENDING"}

2Latency vs throughputBasics

Latency is how long one request takes; throughput is how many requests the system finishes per second. You can raise throughput with more machines, but latency is bounded by the slowest hop on the request path.

• Quote percentiles (p50, p99), not averages
• Memory ~100 ns, SSD ~100 µs, same-region network ~0.5 ms, cross-continent ~150 ms
• Batching improves throughput but often hurts latency

3Vertical vs horizontal scalingScale

Vertical scaling buys a bigger machine: simple, but capped and a single point of failure. Horizontal scaling adds more machines behind a load balancer: near-unlimited and fault tolerant, but forces stateless services and distributed data.

• Start vertical, go horizontal when you hit limits
• Move session state to Redis or tokens so any server can serve any user
• Databases are the hard part to scale out

4Back-of-envelope estimationInterview

Quick arithmetic that sizes the system before you draw it: requests per second, storage per year, bandwidth and cache size. Round aggressively; the goal is the order of magnitude that decides whether you need sharding or a cache.

• 1 day ≈ 86,400 s ≈ 10^5 s
• Peak QPS ≈ 2-3x average
• State assumptions out loud before multiplying

URL shortener: 100M new URLs/day, read:write = 10:1
Write QPS  = 100M / 10^5 s      ≈ 1,000/s   (peak ~3,000/s)
Read QPS   = 10 x 1,000         ≈ 10,000/s  (peak ~30,000/s)
Storage    = 100M x 500 B/day   = 50 GB/day
5 years    = 50 GB x 365 x 5    ≈ 90 TB  -> needs sharding
Cache 20% of daily reads: 0.2 x 1B x 500 B ≈ 100 GB RAM

5Load balancingScale

A load balancer spreads incoming requests across healthy servers, removes failed ones via health checks and lets you add capacity without clients noticing. L4 balancers route by IP/port; L7 balancers can route by URL, header or cookie.

• Algorithms: round robin, least connections, IP/consistent hash
• Avoid sticky sessions; keep servers stateless
• Run the balancer itself in an active-passive pair

6CachingPerformance

Keep frequently read data in fast memory such as Redis so most reads skip the database. Cache-aside is the common pattern: read the cache, on a miss load from the DB and populate it with a TTL.

• Eviction: LRU or LFU when memory is full
• Invalidation: delete the key on write, or rely on short TTLs
• Watch for stampedes on hot keys; use locks or jittered TTLs

read(key):
  v = redis.GET(key)
  if v != null: return v              # cache hit
  v = db.query(key)                   # cache miss
  redis.SET(key, v, EX=300)           # TTL 5 min
  return v
write(key, v):
  db.update(key, v); redis.DEL(key)   # invalidate, next read refills

7CDNPerformance

A content delivery network caches static and media files on edge servers near users, cutting latency and offloading the origin. Ideal for images, JS/CSS bundles and video segments.

• Pull CDN fetches from origin on first miss; push CDN is pre-loaded
• Bust caches with versioned file names (app.3f9c.js)
• Signed URLs protect paid or private content

8Consistent hashingDistribution

Places servers and keys on a hash ring; each key belongs to the next server clockwise. Adding or removing a node moves only about 1/N of the keys instead of nearly all of them, as hash % N would.

• Virtual nodes smooth out uneven load
• Used by DynamoDB, Cassandra, CDNs and cache clusters

import java.util.*;
class HashRing {
    private final TreeMap<Integer, String> ring = new TreeMap<>();
    void addNode(String node, int vnodes) {
        for (int i = 0; i < vnodes; i++) ring.put((node + "#" + i).hashCode(), node);
    }
    String nodeFor(String key) {
        var e = ring.ceilingEntry(key.hashCode());
        return (e != null ? e : ring.firstEntry()).getValue();
    }
}

9API gatewayEdge

A single entry point in front of your services that handles cross-cutting concerns: authentication, rate limiting, routing, request shaping, TLS termination and logging. Services behind it stay focused on business logic.

• Examples: Kong, AWS API Gateway, NGINX, Envoy
• Can aggregate several service calls for one client screen (BFF)
• Keep it thin; it must not become a bottleneck or a monolith

10ReplicationData

Keep copies of data on several nodes for availability and read scale. In leader-follower setups all writes go to the leader and stream to read replicas, which may lag slightly behind.

• Replication lag means a user may not see their own write; read-your-writes by routing to the leader
• Sync replication = safer but slower; async = faster, may lose recent writes
• Failover promotes a follower when the leader dies

11ShardingData

Split one big dataset across many databases by a partition key, so each shard holds only part of the data and traffic. The key choice decides everything: it must spread load evenly and keep common queries on one shard.

• Hash sharding spreads evenly; range sharding keeps ranges together
• Hot shards: a celebrity user or today's date can overload one shard
• Cross-shard joins and transactions become expensive

shard_id = hash(user_id) % NUM_SHARDS      # simple, but resharding moves most keys
shard_id = ring.lookup(hash(user_id))      # consistent hashing, moves ~1/N keys
Bad key:  created_date  -> all of today's writes hit one shard
Good key: user_id       -> even spread, a user's data stays together

12CAP theoremTheory

When a network partition happens, a distributed store must choose between consistency (every read sees the latest write) and availability (every request gets a response). Partitions are unavoidable, so the real choice is CP vs AP.

• CP: banking, inventory (e.g. HBase, Spanner-style)
• AP: feeds, carts, likes (e.g. Cassandra, DynamoDB defaults)
• PACELC adds: without partitions, trade latency vs consistency

13Unique ID generationData

Distributed systems need IDs without a central counter. UUIDv4 is random and collision-safe but large and unsortable; Snowflake-style IDs pack a timestamp, machine ID and sequence into 64 bits, so they are compact and roughly time-ordered.

Snowflake 64-bit ID
| 1 bit sign | 41 bits ms timestamp | 10 bits machine id | 12 bits sequence |
41 bits of ms ≈ 69 years, 4,096 IDs per ms per machine
UUIDv4: 128 bits, random, e.g. 3f1c9a2e-8b7d-4c1e-9a6f-2d4b8e7c1a05

14Blob storageStorage

Object stores like Amazon S3 hold large binary files (images, videos, backups) cheaply and durably, addressed by bucket and key. Keep only metadata in your database and let clients upload or download directly using short-lived presigned URLs.

• Presigned URLs keep large files off your app servers
• Use multipart upload for big files
• Pair with a CDN for downloads

aws s3 presign s3://user-uploads/avatars/42.png --expires-in 600
# -> https://user-uploads.s3.amazonaws.com/avatars/42.png?X-Amz-Signature=...

15Message queuesAsync

A queue or log like Kafka, RabbitMQ or SQS lets producers hand off work and move on, while consumers process it at their own pace. This absorbs traffic spikes, decouples services and lets one event fan out to many subscribers (pub/sub).

• Most systems deliver at-least-once, so consumers must handle duplicates
• Kafka keeps ordering per partition, not across a topic
• Send poison messages to a dead-letter queue after N retries

16WebSockets vs pollingReal-time

Short polling asks the server repeatedly; long polling holds each request open until data arrives; Server-Sent Events stream one-way server-to-client updates; WebSockets give a persistent two-way channel for chat, games and live collaboration.

• SSE is enough for feeds, scores and notifications
• WebSocket servers are stateful; track which server holds each user
• Fall back to long polling behind strict proxies

17Monolith vs microservicesArchitecture

A monolith is one deployable app: simple to build, test and debug. Microservices split the system into independently deployable services owned by separate teams, gaining independent scaling and releases at the cost of network calls, distributed data and ops overhead.

• Start with a modular monolith; split when teams or scaling demand it
• Each service owns its database
• Needs service discovery, tracing and good CI/CD

18Rate limitingProtection

Caps how many requests a client may make in a time window to protect services from abuse and overload. Token bucket allows short bursts while enforcing an average rate; sliding window counters give smoother limits than fixed windows.

• Return HTTP 429 with a Retry-After header
• Keep counters in Redis for a fleet-wide limit
• Limit per user, per IP and per API key

class TokenBucket {
    private final long capacity; private final double refillPerSec;
    private double tokens; private long last = System.nanoTime();
    TokenBucket(long cap, double rate) { capacity = cap; refillPerSec = rate; tokens = cap; }
    synchronized boolean tryAcquire() {
        long now = System.nanoTime();
        tokens = Math.min(capacity, tokens + (now - last) / 1e9 * refillPerSec);
        last = now;
        if (tokens < 1) return false; tokens--; return true;
    }
}

19IdempotencyCorrectness

An operation is idempotent if repeating it has the same effect as doing it once. Clients send a unique idempotency key with retries; the server stores the first result for that key and returns it again instead of charging twice.

• GET, PUT and DELETE should be idempotent; POST usually is not
• Store key + response with a TTL, guarded by a unique constraint
• Essential for payments, orders and queue consumers

curl -X POST https://api.example.com/v1/payments \
  -H "Idempotency-Key: 7c1e9a2f-0b4d-4e7a-9f1c-55d2a8e3b6c0" \
  -d '{"amount": 49900, "currency": "INR"}'
# retry with the same key -> same payment returned, no double charge

20Monitoring, logging & alertsOps

Observability rests on metrics (numbers over time), logs (detailed events) and traces (one request's path across services). Define SLOs such as 99.9% of requests under 300 ms and alert on symptoms users feel, not on every CPU spike.

• Golden signals: latency, traffic, errors, saturation
• Structured JSON logs with a request/trace ID
• Tools: Prometheus + Grafana, ELK, OpenTelemetry, PagerDuty

21How to run a design interviewInterview

Follow a steady sequence so you never freeze: clarify requirements, estimate scale, define the API, sketch the data model, draw the high-level design, deep-dive into the hardest parts, then discuss trade-offs and bottlenecks.

• Spend ~5 minutes on functional and non-functional requirements
• Name the trade-off every time you pick a component
• Finish with failure modes, monitoring and what you would do next

1. Requirements   functional + non-functional (scale, latency, consistency)
2. Estimates      QPS, storage, bandwidth
3. API            key endpoints and payloads
4. Data model     entities, SQL vs NoSQL, partition key
5. High level     clients -> LB -> services -> cache/DB/queue
6. Deep dives     the 1-2 hardest components
7. Trade-offs     bottlenecks, failures, what you'd improve