Design a Notification System

Requirements

A notification system lets internal services (orders, payments, social features, marketing) send messages to users over several channels without each service integrating with Apple, Google, an SMS gateway and an email provider on its own.

Functional

• Send notifications over mobile push (APNs, FCM), SMS, email and in-app (inbox / real-time badge).
• Support transactional messages ("your OTP is 4821") and bulk/marketing campaigns ("50% off today").
• Templates with variables and localization.
• Per-user preferences: opt in or out per channel and per category, quiet hours, time zone.
• Scheduled delivery and delivery status tracking (sent, delivered, opened, failed).

Non-functional

• Reliable: at-least-once delivery; no notification silently lost.
• No spamming: deduplicate repeats and cap how many messages a user receives.
• Low latency for critical messages: OTPs and security alerts in seconds, even during a marketing blast.
• Scalable: handle bursts of tens of millions of messages in a campaign.
• Pluggable providers: swap or fail over between SMS/email vendors.

Capacity Estimates

Assume 100 M daily active users who receive on average 10 notifications per day across all channels.

The raw throughput is modest for a queue-based system; the hard parts are third-party provider limits, retries, prioritization and user-level throttling.

API Design

POST /api/v1/notifications
  Headers: Idempotency-Key: 6f1c...           (required)
  Body: {
    "userIds": ["u_123"],                      // or "segmentId" for campaigns
    "category": "order_update",                // drives preferences and priority
    "templateId": "order_shipped_v3",
    "params": { "orderId": "A-991", "eta": "Tuesday" },
    "channels": ["push", "email"],             // optional; default from category
    "priority": "high",                        // high | normal | low
    "sendAt": "2026-11-01T09:00:00Z"           // optional
  }
  202: { "notificationId": "n_8f2..." }

GET  /api/v1/notifications/{id}               -> per-channel status
PUT  /api/v1/users/{userId}/preferences       -> { "marketing": { "email": false }, "quietHours": "22:00-07:00" }
POST /api/v1/devices                          -> { "userId": "...", "platform": "ios", "token": "..." }
GET  /api/v1/users/{userId}/inbox?cursor=...  -> in-app notification feed

202 Accepted signals the request is queued, not delivered. The idempotency key lets a caller safely retry the POST without sending the message twice.

Data Model

CREATE TABLE deliveries (
  notification_id  TEXT,
  channel          TEXT,          -- push | sms | email | inapp
  status           TEXT,          -- queued | sent | delivered | failed | suppressed
  provider         TEXT,
  provider_msg_id  TEXT,
  attempts         INT,
  updated_at       TIMESTAMP,
  PRIMARY KEY (notification_id, channel)
);

High-Level Design

flowchart LR
  S[Producer Services] --> API[Notification API]
  API --> IDS[(Idempotency Store)]
  API --> NQ[[Inbound Queue]]
  SCH[Scheduler] --> NQ
  NQ --> PR[Processor]
  PR --> PREF[(Preferences and Devices)]
  PR --> TPL[(Templates)]
  PR --> RL[User Rate Limiter]
  PR --> PQ[[Push Queue]]
  PR --> EQ[[Email Queue]]
  PR --> SQ[[SMS Queue]]
  PR --> IQ[[In-app Queue]]
  PQ --> PW[Push Workers] --> APNS[APNs and FCM]
  EQ --> EW[Email Workers] --> ESP[Email Provider]
  SQ --> SW[SMS Workers] --> SMS[SMS Gateway]
  IQ --> IW[In-app Workers] --> WS[WebSocket Gateway]
  PW --> LOG[(Delivery Log)]
  EW --> LOG
  SW --> LOG
  IW --> LOG
  APNS -.->|callbacks| CB[Status Collector] --> LOG

Flow:

1. A producer calls the API with an idempotency key. The API validates, records the key, persists the notification and enqueues it, returning 202.
2. The processor expands segments into users, loads preferences and device tokens, drops channels the user opted out of, applies quiet hours and per-user rate limits, renders templates per locale, and publishes one message per channel to that channel's queue.
3. Channel workers call the external provider, record the outcome, and retry or dead-letter failures.
4. Provider callbacks (delivery receipts, bounces, invalid tokens) update the delivery log and clean up dead device tokens.

Putting a queue between the API and the workers, and another per channel, is the core idea: a slow SMS vendor cannot back up push delivery, and a traffic spike becomes a longer queue rather than an outage.

Deep Dives

Reliability: retries, idempotency and dead letters

• At-least-once: workers acknowledge a queue message only after the provider accepts it. A worker crash means redelivery, never loss.
• Retries with exponential backoff and jitter for transient errors (timeouts, 5xx, 429). Permanent errors (invalid token, unsubscribed email, invalid phone number) are not retried.
• Dead-letter queue after N attempts, with alerting and a replay tool.
• Deduplication: at-least-once means duplicates are possible. Each channel message carries a deterministic ID (notificationId + channel + userId); the worker checks a short-TTL dedupe store (Redis SET NX) before sending. Providers such as FCM and APNs also accept collapse identifiers so a newer message replaces an older one on the device.

Prioritization: keep OTPs fast during a blast

A 50-million-user marketing campaign must not delay a password-reset code. Options:

• Separate queues per priority (high, normal, bulk) with dedicated worker pools, or weighted consumption that favours the high queue.
• Separate provider quotas: reserve part of the SMS/email sending rate for transactional traffic.
• Campaign pacing: the scheduler releases bulk sends at a controlled rate and spreads them by recipient time zone.

User preferences, quiet hours and anti-spam

• Cache preferences aggressively (read on every message; change rarely). Invalidate on update.
• Regulatory rules are hard constraints: unsubscribe links in marketing email, SMS opt-out keywords, consent records.
• Per-user frequency caps (for example at most 3 marketing pushes per day) via a sliding-window counter in Redis.
• Aggregation: collapse bursts ("Alex and 12 others liked your post") by buffering low-priority social notifications for a short window before rendering.
• Quiet hours defer non-critical messages to the scheduler instead of dropping them.

Talking to third-party providers

• Push: APNs (Apple) and FCM (Google) require a valid device token per installation. Remove tokens the provider reports as unregistered; otherwise you waste capacity and risk throttling.
• Rate limits: each provider caps throughput. Workers use a token bucket per provider account and per destination.
• Failover: keep two SMS/email vendors behind a common interface; a circuit breaker routes traffic to the backup when error rates spike.
• Connection reuse: APNs uses long-lived HTTP/2 connections; open a pool and multiplex rather than reconnecting per message.

In-app notifications and status tracking

In-app messages are written to the user's inbox table and pushed over an existing WebSocket/SSE connection if the user is online; otherwise they wait in the inbox and the badge count is computed on next open. Delivery events (sent, delivered, opened, clicked, bounced) flow through a stream into an analytics store for dashboards and for tuning send-time and frequency.

Scaling & Reliability

• Stateless API, processors and workers that autoscale on queue depth, per channel.
• Kafka (or SQS/RabbitMQ) partitioned by userId keeps one user's messages ordered and lets a consumer group scale horizontally.
• Fan-out for campaigns: expand segments in batches (for example 1,000 users per message) to avoid one giant job; checkpoint progress so a crash resumes mid-campaign.
• Storage: delivery logs are append-heavy and time-ordered, a good fit for Cassandra or similar; set TTLs for old records.
• Multi-region: run the pipeline in several regions; users are homed to one region so preferences and dedupe stay local.
• Observability: per-channel success rate, provider latency, queue lag, DLQ size, and end-to-end time for high-priority messages.

Trade-offs

Interview Tips

• Draw the queue between API and workers first; nearly every follow-up (spikes, slow providers, retries) is answered by it.
• Separate concerns: validation, preference/rate-limit processing, channel delivery, status tracking.
• Say "at-least-once with idempotency" explicitly and show where the keys live.
• Explain how critical messages bypass bulk traffic.
• Mention device-token hygiene, provider rate limits, and SMS cost; they show production experience.
• Do not forget the user: preferences, quiet hours, frequency caps and unsubscribe compliance.

References

• System Design Primer: Asynchronism (message and task queues, back pressure)
• ByteByteGo: System Design 101
• Firebase Cloud Messaging documentation
• Apple: Sending notification requests to APNs
• Karan Pratap Singh: System Design