System Design: Must-Know Concepts

1Client-server & APIsBasics

Clients send requests over the network to servers that own data and logic. The API is the contract between them: REST over HTTP is the default, gRPC for fast service-to-service calls, GraphQL when clients need flexible reads.

Under the hood a client opens a TCP (and usually TLS) connection, sends an HTTP request with a method, path, headers and body, and the server replies with a status code and payload. REST keeps each request stateless, so any server behind a load balancer can answer it. gRPC sends compact Protobuf messages over HTTP/2 and suits internal calls; GraphQL lets a client fetch exactly the fields one screen needs in a single round trip.

Real example: Netflix exposes public APIs to devices while its internal microservices talk mostly over gRPC for speed and strict contracts.

Common follow-ups: How would you paginate a feed (offset vs cursor)? How do you evolve an API without breaking old mobile apps? How do you authenticate calls (session cookie, JWT, API key)? Be ready to write two or three endpoints with request and response shapes.

sequenceDiagram
    participant C as Client
    participant S as API server
    participant DB as Database
    C->>S: POST /v1/orders
    S->>DB: INSERT order
    DB-->>S: ok
    S-->>C: 201 Created
    C->>S: GET /v1/orders/9f2
    S->>DB: SELECT order
    DB-->>S: row
    S-->>C: 200 OK + JSON

• Use nouns + HTTP verbs: GET /users/42, POST /orders
• Version APIs (/v1/) and paginate list endpoints
• Status codes matter: 2xx ok, 4xx caller error, 5xx server error

curl -X POST https://api.example.com/v1/orders \
  -H "Content-Type: application/json" \
  -d '{"itemId": 17, "qty": 2}'
# -> 201 Created  {"orderId": "ord_9f2", "status": "PENDING"}

2Latency vs throughputBasics

Latency is how long one request takes; throughput is how many requests the system finishes per second. You can raise throughput with more machines, but latency is bounded by the slowest hop on the request path.

Think of a highway: latency is how long one car takes to drive from A to B, throughput is how many cars pass per hour. Adding lanes (servers) raises throughput, but the trip still takes as long as the slowest stretch of road. Queues link the two: as utilisation nears 100%, requests wait longer and tail latency explodes.

That is why teams track p99 latency. Amazon famously found that small delays cost real sales, and at large fan-out even a rare slow call becomes common: if one page touches 100 services, most page loads will hit at least one p99-slow call. Techniques like caching, parallel calls and request hedging cut latency; batching and async queues lift throughput.

Interviewers often ask: Which matters more for this system? How would you cut p99? What happens to latency as load approaches capacity?

flowchart TB
    subgraph L["Latency: time for one request"]
        U([Client]) ==>|"10 ms"| A[API]
        A ==>|"5 ms"| W[Worker]
        W ==>|"30 ms"| DB[(DB)]
    end
    subgraph T["Throughput: requests per second"]
        A2["API<br/>5,000/s"] ==> W2["Worker<br/>800/s"]
        W2 ==> D2[("DB<br/>2,000/s")]
    end
    L ~~~ T

• Quote percentiles (p50, p99), not averages
• Memory ~100 ns, SSD ~100 µs, same-region network ~0.5 ms, cross-continent ~150 ms
• Batching improves throughput but often hurts latency

3Vertical vs horizontal scalingScale

Vertical scaling buys a bigger machine: simple, but capped and a single point of failure. Horizontal scaling adds more machines behind a load balancer: near-unlimited and fault tolerant, but forces stateless services and distributed data.

Vertical scaling means more CPU, RAM or faster disks on one box. It needs no code changes and keeps transactions simple, which is why many startups run a single large Postgres server for years. The ceiling is the biggest machine you can buy, and that machine is a single point of failure.

Horizontal scaling adds identical machines behind a load balancer. It only works cleanly when app servers are stateless: sessions live in Redis or signed tokens, files live in object storage. Stack Overflow is a well-known example of running huge traffic on a few powerful servers, while Netflix scales out thousands of stateless instances on AWS with auto-scaling groups.

Expect follow-ups like: How do you scale the database once app servers are stateless? When would you choose vertical on purpose? How does auto-scaling decide when to add nodes?

flowchart LR
    subgraph V["Vertical: scale up"]
        U1([Clients]) ==> BIG["One big server<br/>64 CPU, 512 GB"]
    end
    subgraph H["Horizontal: scale out"]
        U2([Clients]) ==> LB[Load balancer]
        LB --> S1[Server 1]
        LB --> S2[Server 2]
        LB --> S3[Server 3]
    end
    V ~~~ H

• Start vertical, go horizontal when you hit limits
• Move session state to Redis or tokens so any server can serve any user
• Databases are the hard part to scale out

4Back-of-envelope estimationInterview

Quick arithmetic that sizes the system before you draw it: requests per second, storage per year, bandwidth and cache size. Round aggressively; the goal is the order of magnitude that decides whether you need sharding or a cache.

The goal is not precision but a defensible order of magnitude that drives design choices. A typical flow: daily active users × actions per user ÷ 10^5 seconds gives average QPS; multiply by 2-3 for peak. Then object size × writes per day × 365 × years gives storage, and QPS × response size gives bandwidth.

Example: 100M daily users posting 2 photos of 500 KB each is 100 TB of new data per day, which instantly tells you the photos belong in blob storage behind a CDN, not in a relational table. Likewise, 10K reads per second with a hot working set of 20 GB fits comfortably in a Redis cache.

Interviewers usually ask: What is your read-to-write ratio? How much cache memory do you need? Does this fit on one machine? Keep numbers round, write them on the board and sanity-check each step.

flowchart TB
    A["10M DAU"] -->|"x 10 per user"| B["100M writes/day"]
    B -->|"/ 10^5 s"| C["~1,000 QPS avg"]
    C -->|"x 3"| D["~3,000 peak QPS"]
    B -->|"x 500 B each"| E["50 GB/day"]
    E -->|"x 365 x 5"| F[("~90 TB in 5 yrs")]

• 1 day ≈ 86,400 s ≈ 10^5 s
• Peak QPS ≈ 2-3x average
• State assumptions out loud before multiplying

URL shortener: 100M new URLs/day, read:write = 10:1
Write QPS  = 100M / 10^5 s      ≈ 1,000/s   (peak ~3,000/s)
Read QPS   = 10 x 1,000         ≈ 10,000/s  (peak ~30,000/s)
Storage    = 100M x 500 B/day   = 50 GB/day
5 years    = 50 GB x 365 x 5    ≈ 90 TB  -> needs sharding
Cache 20% of daily reads: 0.2 x 1B x 500 B ≈ 100 GB RAM

5Load balancingScale

A load balancer spreads incoming requests across healthy servers, removes failed ones via health checks and lets you add capacity without clients noticing. L4 balancers route by IP/port; L7 balancers can route by URL, header or cookie.

An L4 balancer forwards TCP/UDP packets based only on IP and port, so it is fast and cheap. An L7 balancer terminates HTTP, which lets it route /api and /images to different pools, add headers, retry failed calls and do TLS offload. Health checks continuously probe servers and pull unhealthy ones out of rotation.

Choosing an algorithm matters: round robin is fine for uniform requests, least connections helps when some calls are slow, and consistent hashing keeps a user or key pinned to the same backend for cache locality. Large systems layer them: DNS or anycast spreads users across regions, then L4 and L7 balancers spread work inside each region, as Google and Cloudflare do.

Follow-ups to expect: What if the load balancer itself fails? How do you drain a server during deployment? How do you handle WebSocket connections that stay open for hours?

flowchart LR
    U([Clients]) ==> LB[Load balancer]
    LB ==>|"round robin"| S1[Server 1]
    LB ==>|"round robin"| S2[Server 2]
    LB -.-x|"health check failed"| S3[Server 3]

• Algorithms: round robin, least connections, IP/consistent hash
• Avoid sticky sessions; keep servers stateless
• Run the balancer itself in an active-passive pair

6CachingPerformance

Keep frequently read data in fast memory such as Redis so most reads skip the database. Cache-aside is the common pattern: read the cache, on a miss load from the DB and populate it with a TTL.

Caches sit at many layers: browser, CDN, application memory and a shared store like Redis or Memcached. Besides cache-aside, know write-through (write to cache and DB together, fresher but slower writes) and write-back (write to cache first, flush later, fast but risky on crashes).

The hard parts are invalidation and hot keys. Facebook runs massive Memcached clusters in front of MySQL and uses leases to stop thundering herds when a popular key expires. Twitter caches precomputed home timelines in Redis so most reads never touch a database. A healthy cache often serves 90%+ of reads, which dramatically shrinks database load.

Common follow-ups: What do you cache and for how long? How do you keep the cache consistent after an update? What happens if the cache cluster goes down and every request hits the database at once (cache stampede)?

flowchart TB
    U([Client]) ==> A[App server]
    A ==>|"1. GET key, hit ~1 ms"| C{{Redis cache}}
    A -->|"2. miss: query ~20 ms"| DB[(Database)]
    A -.->|"3. SET key with TTL"| C

• Eviction: LRU or LFU when memory is full
• Invalidation: delete the key on write, or rely on short TTLs
• Watch for stampedes on hot keys; use locks or jittered TTLs

read(key):
  v = redis.GET(key)
  if v != null: return v              # cache hit
  v = db.query(key)                   # cache miss
  redis.SET(key, v, EX=300)           # TTL 5 min
  return v
write(key, v):
  db.update(key, v); redis.DEL(key)   # invalidate, next read refills

7CDNPerformance

A content delivery network caches static and media files on edge servers near users, cutting latency and offloading the origin. Ideal for images, JS/CSS bundles and video segments.

When a user requests a file, DNS or anycast routing sends them to the nearest edge location. If the edge has a fresh copy it serves it immediately; otherwise it fetches from the origin (or a regional mid-tier cache), stores it per the Cache-Control headers and serves it. Hit ratio is the key metric, since every miss travels back to your servers.

Netflix built its own CDN, Open Connect, placing appliances inside ISP networks so most video bytes never cross the wider internet. Modern CDNs also run code at the edge, terminate TLS close to users and absorb DDoS attacks.

Interview follow-ups: How do you push an urgent update when files are cached for a day (purge vs versioned URLs)? Can a CDN cache API responses or personalised pages? How would you protect premium video from hot-linking? Mention signed URLs and short TTLs.

flowchart TB
    U1([User in India]) ==>|"~20 ms"| E1{{Edge Mumbai}}
    U2([User in US]) ==>|"~20 ms"| E2{{Edge Virginia}}
    E1 -.->|"miss only"| O[Origin server]
    E2 -.->|"miss only"| O
    O --> S[(Blob storage)]

• Pull CDN fetches from origin on first miss; push CDN is pre-loaded
• Bust caches with versioned file names (app.3f9c.js)
• Signed URLs protect paid or private content

8Consistent hashingDistribution

Places servers and keys on a hash ring; each key belongs to the next server clockwise. Adding or removing a node moves only about 1/N of the keys instead of nearly all of them, as hash % N would.

Both servers and keys are hashed onto the same circular space, say 0 to 2^32. To find a key's owner you walk clockwise to the first server point. With plain hash % N, changing N from 10 to 11 remaps about 90% of keys, wiping your cache; on a ring only the keys between the new node and its neighbour move.

A raw ring can be lumpy, so each physical server gets many virtual nodes scattered around it, which evens out load and lets stronger machines take more points. Amazon's Dynamo paper popularised the idea, Cassandra uses token rings, and Discord used a ring to route each chat guild to a consistent process.

Common follow-ups: How do you handle a hot key that one node cannot serve alone? How does replication work on the ring (copy to the next N nodes)? What alternatives exist, such as rendezvous hashing or a fixed slot map like Redis Cluster's 16,384 hash slots?

flowchart TB
    N1["Node A<br/>pos 0"] -->|"clockwise"| N2["Node B<br/>pos 120"]
    N2 -->|"clockwise"| N3["Node C<br/>pos 240"]
    N3 -->|"wraps to 0"| N1
    K1("k1 at 50") -.->|"owner"| N2
    K2("k2 at 200") -.->|"owner"| N3
    K3("k3 at 300") -.->|"owner"| N1

• Virtual nodes smooth out uneven load
• Used by DynamoDB, Cassandra, CDNs and cache clusters

import java.util.*;
class HashRing {
    private final TreeMap<Integer, String> ring = new TreeMap<>();
    void addNode(String node, int vnodes) {
        for (int i = 0; i < vnodes; i++) ring.put((node + "#" + i).hashCode(), node);
    }
    String nodeFor(String key) {
        var e = ring.ceilingEntry(key.hashCode());
        return (e != null ? e : ring.firstEntry()).getValue();
    }
}

9API gatewayEdge

A single entry point in front of your services that handles cross-cutting concerns: authentication, rate limiting, routing, request shaping, TLS termination and logging. Services behind it stay focused on business logic.

A gateway is a reverse proxy with extra brains. A request arrives, the gateway validates the token, checks rate limits, picks the right backend from a routing table, maybe transforms the payload, then forwards it and records metrics. Doing this once at the edge stops every team from re-implementing auth and throttling.

A related pattern is Backend for Frontend (BFF): a separate gateway per client type, so the mobile app gets a slim aggregated response while the web app gets richer data. Netflix built its Zuul gateway to front all device traffic, and it also lets them shift users between AWS regions during failures.

Interviewers may ask how a gateway differs from a load balancer (gateways understand API semantics; balancers just spread load), how to avoid it becoming a single point of failure (run many stateless instances behind a balancer), and whether business logic belongs there (it should not).

flowchart LR
    W([Web app]) ==> G["API gateway<br/>auth, rate limit"]
    M([Mobile app]) ==> G
    G -->|"/users"| US[User service]
    G -->|"/orders"| OS[Order service]
    G -->|"/payments"| PS[Payment service]

• Examples: Kong, AWS API Gateway, NGINX, Envoy
• Can aggregate several service calls for one client screen (BFF)
• Keep it thin; it must not become a bottleneck or a monolith

10ReplicationData

Keep copies of data on several nodes for availability and read scale. In leader-follower setups all writes go to the leader and stream to read replicas, which may lag slightly behind.

In leader-follower replication the leader writes each change to a log (MySQL binlog, Postgres WAL) and followers replay it in order. Synchronous replication waits for a follower to confirm before acknowledging, trading latency for durability; asynchronous acknowledges immediately, so a crash can lose the last few writes. Many setups use one sync follower plus several async ones.

Multi-leader replication lets each region accept writes, which helps global apps but needs conflict resolution. Leaderless systems like Cassandra and DynamoDB write to several replicas and use quorums (W + R > N) for consistency. Instagram, for example, has long relied on read replicas to serve its very read-heavy traffic.

Typical follow-ups: How do you guarantee read-your-writes? What happens if two nodes both think they are leader (split brain)? How is failover detected and who promotes the new leader?

flowchart TB
    U([Client]) ==> A[App server]
    A ==>|"writes"| L[(Leader DB)]
    L -.->|"async copy"| F1[(Follower 1)]
    L -.->|"async copy"| F2[(Follower 2)]
    A -->|"reads"| F1
    A -->|"reads"| F2

• Replication lag means a user may not see their own write; read-your-writes by routing to the leader
• Sync replication = safer but slower; async = faster, may lose recent writes
• Failover promotes a follower when the leader dies

11ShardingData

Split one big dataset across many databases by a partition key, so each shard holds only part of the data and traffic. The key choice decides everything: it must spread load evenly and keep common queries on one shard.

A router or the client library maps each key to a shard, either by hashing it (even spread, but range scans hit every shard), by range (easy range queries, risk of hot ranges) or via a lookup directory (flexible, but one more component to keep available). Resharding later is painful, so many teams pre-split into many logical shards mapped onto fewer physical machines.

Instagram shards Postgres by user ID across thousands of logical shards, and Discord partitions messages by channel plus a time bucket so one busy channel's history stays together but bounded in size. Pair sharding with replication so each shard also survives node failures.

Interviewers like to ask: What is your shard key and why? How do you query across shards, for example 'top posts globally'? How do you handle a celebrity hot spot? How would you add shards with no downtime?

flowchart TB
    A[App server] ==> R[Shard router]
    R ==> H{"hash(user_id) % 3"}
    H -->|"0"| S0[(Shard 0)]
    H -->|"1"| S1[(Shard 1)]
    H -->|"2"| S2[(Shard 2)]

• Hash sharding spreads evenly; range sharding keeps ranges together
• Hot shards: a celebrity user or today's date can overload one shard
• Cross-shard joins and transactions become expensive

shard_id = hash(user_id) % NUM_SHARDS      # simple, but resharding moves most keys
shard_id = ring.lookup(hash(user_id))      # consistent hashing, moves ~1/N keys
Bad key:  created_date  -> all of today's writes hit one shard
Good key: user_id       -> even spread, a user's data stays together

12CAP theoremTheory

When a network partition happens, a distributed store must choose between consistency (every read sees the latest write) and availability (every request gets a response). Partitions are unavoidable, so the real choice is CP vs AP.

Picture two data centres that lose their link. A write lands in one. If the other still answers reads, it may return stale data (AP); if it refuses until the link heals, it stays correct but unavailable (CP). Without a partition you can have both, which is why CAP is really about behaviour during failures.

Real systems choose per use case. A bank ledger or ticket booking prefers CP, since showing a wrong balance or double-selling a seat is worse than a brief error. A social feed or 'like' counter prefers AP and reconciles later. Amazon's shopping cart is the classic AP example: it would rather merge two cart versions than reject an add-to-cart click.

Follow-ups: Which parts of your design need strong consistency and which can be eventual? What is PACELC? How do quorum settings in Cassandra or DynamoDB move you along the spectrum?

flowchart TB
    U([Client write]) ==> R[Replica A]
    R ==> D{Partition from<br/>Replica B?}
    D -->|"no"| OK["Serve normally<br/>C and A"]
    D -->|"yes, choose CP"| CP["Reject write<br/>stay consistent"]
    D -->|"yes, choose AP"| AP["Accept write<br/>reads may be stale"]

• CP: banking, inventory (e.g. HBase, Spanner-style)
• AP: feeds, carts, likes (e.g. Cassandra, DynamoDB defaults)
• PACELC adds: without partitions, trade latency vs consistency

13Unique ID generationData

Distributed systems need IDs without a central counter. UUIDv4 is random and collision-safe but large and unsortable; Snowflake-style IDs pack a timestamp, machine ID and sequence into 64 bits, so they are compact and roughly time-ordered.

Options form a spectrum. A single database auto-increment is simple but a bottleneck and single point of failure. Ticket servers (Flickr used two MySQL boxes handing out odd and even IDs) scale a bit further. UUIDv4 needs no coordination but is 128 bits and random, which fragments B-tree indexes. Snowflake uses 41 bits of milliseconds since a custom epoch, 10 bits of machine ID and 12 bits of sequence, so one node can mint about 4,096 IDs per millisecond.

Twitter created Snowflake for tweets, Discord uses Snowflake IDs for messages and users, and Instagram generates similar IDs inside Postgres per shard. Newer formats such as ULID and UUIDv7 also put the timestamp first so IDs sort by time.

Follow-ups: What happens if a server's clock moves backwards? How do machines get unique worker IDs? Why do time-sortable IDs make pagination and sharding easier?

flowchart LR
    T["41 bits<br/>timestamp ms"] --> ID["64-bit<br/>Snowflake ID"]
    M["10 bits<br/>machine id"] --> ID
    Q["12 bits<br/>sequence"] --> ID
    ID ==>|"time-sortable"| DB[(Primary key)]

Snowflake 64-bit ID
| 1 bit sign | 41 bits ms timestamp | 10 bits machine id | 12 bits sequence |
41 bits of ms ≈ 69 years, 4,096 IDs per ms per machine
UUIDv4: 128 bits, random, e.g. 3f1c9a2e-8b7d-4c1e-9a6f-2d4b8e7c1a05

14Blob storageStorage

Object stores like Amazon S3 hold large binary files (images, videos, backups) cheaply and durably, addressed by bucket and key. Keep only metadata in your database and let clients upload or download directly using short-lived presigned URLs.

An object store keeps immutable objects in a flat namespace of bucket plus key, with metadata and an HTTP API. Internally it splits data into chunks, spreads them across disks and data centres, and uses replication or erasure coding to reach very high durability; Amazon S3 is designed for eleven nines. You cannot edit part of an object in place, only replace it.

Typical upload flow: the client asks your API for a presigned URL, uploads straight to S3, and an event notification triggers processing such as thumbnail generation or video transcoding. Instagram, Dropbox and YouTube-style systems all store media this way and keep only references in their databases.

Interview follow-ups: How do you handle a 5 GB upload over a flaky network (multipart, resumable)? How do you serve files quickly worldwide (CDN)? How do you lower storage cost for old data (lifecycle rules to cold tiers)?

flowchart TB
    U([Client]) -->|"1. ask to upload"| A[API server]
    A -.->|"2. presigned URL"| U
    U ==>|"3. PUT file"| B[(S3 bucket)]
    A -->|"save metadata"| DB[(Metadata DB)]
    B -.->|"downloads"| C{{CDN}}

• Presigned URLs keep large files off your app servers
• Use multipart upload for big files
• Pair with a CDN for downloads

aws s3 presign s3://user-uploads/avatars/42.png --expires-in 600
# -> https://user-uploads.s3.amazonaws.com/avatars/42.png?X-Amz-Signature=...

15Message queuesAsync

A queue or log like Kafka, RabbitMQ or SQS lets producers hand off work and move on, while consumers process it at their own pace. This absorbs traffic spikes, decouples services and lets one event fan out to many subscribers (pub/sub).

Two models matter. Classic queues (SQS, RabbitMQ) hand each message to one consumer and delete it after an ack. Logs (Kafka, Kinesis) append messages to partitions and keep them for days, so many consumer groups can read at their own offsets and even replay history. Ordering is guaranteed within a partition, so choose the partition key carefully, for example by user or order ID.

Uber streams trip and location events through Kafka to power pricing, ETAs and analytics, and LinkedIn built Kafka for its activity feed. Queues also enable the outbox pattern and smooth bursts, like a flash sale writing orders faster than a payment service can process them.

Common follow-ups: How do you avoid processing a message twice? What if consumers fall behind (lag, autoscaling, backpressure)? When do you need exactly-once semantics, and how close can you really get?

flowchart TB
    U([Client]) ==>|"202 Accepted"| P[Order service]
    P ==>|"publish"| Q[[Queue]]
    Q -.->|"consume"| W1[Worker 1]
    Q -.->|"consume"| W2[Worker 2]
    Q -.->|"after N failures"| DLQ[[Dead-letter queue]]

• Most systems deliver at-least-once, so consumers must handle duplicates
• Kafka keeps ordering per partition, not across a topic
• Send poison messages to a dead-letter queue after N retries

16WebSockets vs pollingReal-time

Short polling asks the server repeatedly; long polling holds each request open until data arrives; Server-Sent Events stream one-way server-to-client updates; WebSockets give a persistent two-way channel for chat, games and live collaboration.

A WebSocket starts as an HTTP request with an Upgrade header, then stays open as a full-duplex TCP stream, so either side can send small frames at any time with tiny overhead. Long polling fakes push by having the server hold the request until an event or timeout, then the client immediately reconnects. SSE is a simple one-way HTTP stream with built-in auto-reconnect.

Scaling persistent connections is the real challenge. Each gateway server holds many thousands of open sockets, so you need a registry of which user lives on which server and a pub/sub layer (such as Redis) to deliver a message to the right box. WhatsApp famously held around two million connections on a single Erlang server, and Discord fans messages out through stateful gateway processes.

Expect follow-ups: How do you deliver messages to offline users? What happens when a gateway server restarts? How do you handle heartbeats and reconnect storms?

sequenceDiagram
    participant C as Client
    participant S as Server
    Note over C,S: Short polling
    C->>S: any news?
    S-->>C: no
    C->>S: any news?
    S-->>C: yes, 1 message
    Note over C,S: WebSocket
    C->>S: HTTP Upgrade
    S-->>C: 101 Switching Protocols
    S-)C: push message
    S-)C: push message

• SSE is enough for feeds, scores and notifications
• WebSocket servers are stateful; track which server holds each user
• Fall back to long polling behind strict proxies

17Monolith vs microservicesArchitecture

A monolith is one deployable app: simple to build, test and debug. Microservices split the system into independently deployable services owned by separate teams, gaining independent scaling and releases at the cost of network calls, distributed data and ops overhead.

In a monolith, modules call each other in-process, share one database and ship together, so refactoring and transactions are easy. In microservices those calls become network requests that can time out, so you need retries, circuit breakers, service discovery, distributed tracing and patterns like sagas instead of cross-service transactions.

Amazon and Netflix moved to microservices to let hundreds of teams deploy independently many times a day. On the other hand, Shopify runs a very large modular monolith successfully, and Amazon Prime Video publicly described moving one monitoring pipeline back to a monolith to cut costs. The lesson: architecture should follow team size and scaling needs, not fashion.

Interviewers often ask: Where would you draw service boundaries? How do services share data without sharing a database? How would you migrate a monolith gradually (the strangler fig pattern)?

flowchart LR
    subgraph M["Monolith"]
        APP["One deployable<br/>users + orders"] ==> DB1[(Shared DB)]
    end
    subgraph MS["Microservices"]
        G[API gateway] --> S1[User service]
        G --> S2[Order service]
        S1 --> D1[(Users DB)]
        S2 --> D2[(Orders DB)]
    end
    M ~~~ MS

• Start with a modular monolith; split when teams or scaling demand it
• Each service owns its database
• Needs service discovery, tracing and good CI/CD

18Rate limitingProtection

Caps how many requests a client may make in a time window to protect services from abuse and overload. Token bucket allows short bursts while enforcing an average rate; sliding window counters give smoother limits than fixed windows.

Algorithms to know: fixed window counters are simple but allow double bursts at window edges; sliding window log is exact but memory-heavy; sliding window counter blends two windows cheaply; token bucket refills tokens at a steady rate and allows bursts up to the bucket size; leaky bucket drains requests at a constant rate.

In a fleet, limiters usually live at the API gateway and keep counters in Redis using atomic INCR or a Lua script, so all servers agree. Stripe uses several layers: per-user request limiters plus load shedders that drop low-priority traffic when the system is stressed. GitHub and X return rate-limit headers so clients can back off politely.

Follow-ups: Where do you place the limiter? What if Redis is down (fail open or fail closed)? How do you avoid a race between reading and incrementing a counter? How do you rate limit across regions?

flowchart TB
    U([Client]) ==> G[API gateway]
    G ==> D{Tokens left?}
    B{{"Redis bucket<br/>refill 10/s"}} -.-> D
    D ==>|"yes, take 1"| A[API server]
    D -->|"no"| R["429 + Retry-After"]

• Return HTTP 429 with a Retry-After header
• Keep counters in Redis for a fleet-wide limit
• Limit per user, per IP and per API key

class TokenBucket {
    private final long capacity; private final double refillPerSec;
    private double tokens; private long last = System.nanoTime();
    TokenBucket(long cap, double rate) { capacity = cap; refillPerSec = rate; tokens = cap; }
    synchronized boolean tryAcquire() {
        long now = System.nanoTime();
        tokens = Math.min(capacity, tokens + (now - last) / 1e9 * refillPerSec);
        last = now;
        if (tokens < 1) return false; tokens--; return true;
    }
}

19IdempotencyCorrectness

An operation is idempotent if repeating it has the same effect as doing it once. Clients send a unique idempotency key with retries; the server stores the first result for that key and returns it again instead of charging twice.

Networks fail ambiguously: a client may time out after the server already charged the card. Retrying is only safe if the operation is idempotent. The usual recipe is that the client generates a UUID per logical action and sends it in a header such as Idempotency-Key. The server first inserts that key into a table with a unique constraint, does the work, then saves the response alongside it; a duplicate request just returns the saved response.

Stripe popularised this for payments, and the same idea protects queue consumers that receive at-least-once delivery: store processed message IDs or design writes as upserts so a replay changes nothing.

Interview follow-ups: What if two identical requests arrive at the same moment (lock on the key or rely on the unique index)? How long do you keep keys? What if the same key arrives with a different request body (reject it)?

sequenceDiagram
    participant C as Client
    participant S as Payment API
    participant K as Key store
    C->>S: POST /pay, key abc
    S->>K: save abc = charged
    S--xC: 200 lost in timeout
    C->>S: retry POST /pay, key abc
    S->>K: lookup abc
    K-->>S: already charged
    S-->>C: same 200, no double charge

• GET, PUT and DELETE should be idempotent; POST usually is not
• Store key + response with a TTL, guarded by a unique constraint
• Essential for payments, orders and queue consumers

curl -X POST https://api.example.com/v1/payments \
  -H "Idempotency-Key: 7c1e9a2f-0b4d-4e7a-9f1c-55d2a8e3b6c0" \
  -d '{"amount": 49900, "currency": "INR"}'
# retry with the same key -> same payment returned, no double charge

20Monitoring, logging & alertsOps

Observability rests on metrics (numbers over time), logs (detailed events) and traces (one request's path across services). Define SLOs such as 99.9% of requests under 300 ms and alert on symptoms users feel, not on every CPU spike.

Metrics are cheap aggregated numbers (request rate, error rate, latency histograms) stored in a time-series database and plotted on dashboards. Logs capture rich detail for individual events and are shipped to a central store for search. Distributed traces stitch together the spans of one request across services, so you can see exactly which hop was slow.

Good alerting starts from SLOs and error budgets, a practice popularised by Google SRE: page a human only when users are, or soon will be, affected, and route everything else to tickets or dashboards. Netflix and Uber invest heavily in tracing and anomaly detection because, with thousands of services, nobody can read raw logs during an outage.

Common follow-ups: What would you put on the main dashboard? How do you trace a single slow request? How do you keep logging costs under control at scale (sampling, retention tiers)?

flowchart TB
    S[Services] -->|"metrics"| M[(Prometheus)]
    S -->|"logs"| L[(Log store)]
    S -->|"traces"| T[(Trace store)]
    M --> G[Dashboards]
    L --> G
    T --> G
    M ==> A{SLO breached?}
    A ==>|"yes, page"| O([On-call engineer])

• Golden signals: latency, traffic, errors, saturation
• Structured JSON logs with a request/trace ID
• Tools: Prometheus + Grafana, ELK, OpenTelemetry, PagerDuty

21How to run a design interviewInterview

Follow a steady sequence so you never freeze: clarify requirements, estimate scale, define the API, sketch the data model, draw the high-level design, deep-dive into the hardest parts, then discuss trade-offs and bottlenecks.

A typical 45-minute round can be budgeted roughly like this: about 5 minutes on requirements and scale, 5 on core entities and API, 10-15 on a working high-level design, 15 on deep dives, and the rest on wrap-up. Get something simple working end to end first, then improve it; interviewers reward a clear path more than a perfect diagram.

Drive the conversation. Say which non-functional requirements matter most (for example low latency for reads, strong consistency for payments) and let them guide every choice. During deep dives, pick the riskiest part of the design, such as feed fan-out, hot keys or exactly-once payments, and offer two options with their trade-offs before choosing.

Practise with classic prompts like a URL shortener, chat app, news feed, ride sharing and rate limiter. Talk out loud, check in with the interviewer every few minutes, and treat hints as collaboration rather than as a sign of failure.

flowchart TB
    subgraph S1["Scope, ~10 min"]
        direction LR
        R[1. Requirements] ==> E[2. Estimates]
        E ==> A[3. API]
    end
    subgraph S2["Design, ~25 min"]
        direction LR
        M[4. Data model] ==> H[5. High-level design]
        H ==> DD[6. Deep dives]
    end
    subgraph S3["Wrap up, ~5 min"]
        T[7. Trade-offs]
    end
    S1 ==> S2
    S2 ==> S3

• Spend ~5 minutes on functional and non-functional requirements
• Name the trade-off every time you pick a component
• Finish with failure modes, monitoring and what you would do next

1. Requirements   functional + non-functional (scale, latency, consistency)
2. Estimates      QPS, storage, bandwidth
3. API            key endpoints and payloads
4. Data model     entities, SQL vs NoSQL, partition key
5. High level     clients -> LB -> services -> cache/DB/queue
6. Deep dives     the 1-2 hardest components
7. Trade-offs     bottlenecks, failures, what you'd improve