Design a Payment System

Requirements

Scope this as the payment backend of an e-commerce marketplace: a buyer checks out, we charge their card through a Payment Service Provider (PSP) such as Stripe or Adyen, and later pay out sellers. We do not store raw card numbers ourselves.

Functional

• Pay-in: charge a buyer for an order (authorize, capture, refund).
• Pay-out: transfer money to sellers' bank accounts on a schedule.
• Keep an accurate balance for every account (buyer, seller, platform fees).
• Show payment status to users and support teams.
• Daily reconciliation against PSP and bank settlement files.

Non-functional

• Correctness over latency: never double-charge, never lose a payment. A few hundred ms extra is fine.
• Exactly-once effect despite retries, timeouts and crashes.
• Auditability: every cent must be traceable; records are immutable.
• High availability for checkout (99.99%); outages cost revenue directly.
• Security and compliance: PCI DSS scope minimized, encryption, access controls.

Capacity Estimates

Assume 10 M payment transactions per day, average order value $50, peak traffic 10× average during sales events.

A single well-tuned relational cluster can handle these writes. The difficulty is not throughput; it is that a slow, unreliable external PSP sits in the middle of every transaction.

API Design

POST /v1/payments
  Headers: Idempotency-Key: 7f9c2e1a-...   (generated by client per checkout attempt)
  Body:    { "orderId": "ord_123", "amount": 5000, "currency": "USD",
             "paymentMethodToken": "pm_tok_abc", "buyerId": "u_42" }
  201:     { "paymentId": "pay_88", "status": "PROCESSING" }
  200:     same response replayed if the key was already used
  409:     same key reused with a different body

GET  /v1/payments/{paymentId}
  200:     { "status": "SUCCEEDED", "amount": 5000, "pspRef": "ch_3N..." }

POST /v1/payments/{paymentId}/refunds
  Headers: Idempotency-Key: ...
  Body:    { "amount": 2000, "reason": "item_damaged" }

POST /v1/webhooks/psp        (PSP -> us; signature verified)
  Body:    { "event": "charge.succeeded", "pspRef": "ch_3N...", "ourRef": "pay_88" }

• Amounts are integers in minor units (cents) with an explicit currency; never floating point.
• The card number goes straight from the browser to the PSP's hosted form, which returns a token. Our servers never see the PAN, keeping us mostly out of PCI scope.

Data Model

CREATE TABLE payments (
  payment_id      VARCHAR(32) PRIMARY KEY,
  idempotency_key VARCHAR(64) UNIQUE NOT NULL,
  order_id        VARCHAR(32) NOT NULL,
  buyer_id        BIGINT NOT NULL,
  amount_minor    BIGINT NOT NULL,
  currency        CHAR(3) NOT NULL,
  status          VARCHAR(16) NOT NULL,   -- CREATED, PROCESSING, SUCCEEDED, FAILED, REFUNDED
  psp_reference   VARCHAR(64),
  version         INT NOT NULL,           -- optimistic locking
  created_at      TIMESTAMP, updated_at TIMESTAMP
);

CREATE TABLE ledger_entries (             -- append-only, never updated
  entry_id        BIGINT PRIMARY KEY,
  txn_id          VARCHAR(32) NOT NULL,   -- groups balanced entries
  account_id      VARCHAR(32) NOT NULL,
  direction       CHAR(1) NOT NULL,       -- D or C
  amount_minor    BIGINT NOT NULL CHECK (amount_minor > 0),
  currency        CHAR(3) NOT NULL,
  created_at      TIMESTAMP NOT NULL
);

CREATE TABLE outbox (                     -- transactional outbox for events
  event_id        BIGINT PRIMARY KEY,
  aggregate_id    VARCHAR(32),
  payload         JSON,
  published       BOOLEAN DEFAULT FALSE
);

High-Level Design

flowchart LR
  Client[Checkout Client] --> GW[API Gateway]
  GW --> PS[Payment Service]
  PS --> PDB[(Payments DB + Outbox)]
  PS --> EX[PSP Executor]
  EX --> PSP[External PSP]
  PSP -->|webhook| WH[Webhook Handler]
  WH --> PS
  PDB --> RELAY[Outbox Relay]
  RELAY --> K[[Kafka]]
  K --> LED[Ledger Service]
  K --> WAL[Wallet / Balance Service]
  K --> NOTIF[Notifications]
  LED --> LDB[(Ledger DB)]
  REC[Reconciliation Job] --> LDB
  REC --> FILES[(PSP Settlement Files)]

1. Payment Service checks the idempotency key, creates a payments row in CREATED, and moves it to PROCESSING.
2. The PSP Executor calls the PSP with our payment ID as the PSP's idempotency key.
3. The result arrives synchronously or via webhook. The Payment Service updates status and writes an outbox event in the same DB transaction.
4. The outbox relay publishes to Kafka; the Ledger Service writes balanced entries; wallets and notifications update.
5. Nightly, reconciliation compares our ledger with the PSP's settlement report.

Deep Dives

Idempotency end to end

Retries are everywhere: a user double-clicks, the mobile app retries after a timeout, our service retries the PSP. Each hop needs its own idempotency guarantee.

• Client → us: the client generates an Idempotency-Key per checkout attempt. We insert it under a UNIQUE constraint in the same transaction that creates the payment. A duplicate request hits the constraint, and we return the stored response instead of charging again. If the first request is still in flight, return 409/425 and let the client poll.
• Us → PSP: we pass a deterministic key (our payment_id) to the PSP. If our call times out and we retry, the PSP recognizes the key and returns the original result rather than creating a second charge.
• Event consumers: Kafka is at-least-once, so the ledger stores the processed txn_id with a unique constraint; replays are ignored.

Payment state machine and timeouts

CREATED -> PROCESSING -> SUCCEEDED -> (PARTIALLY_)REFUNDED
                     \-> FAILED
                     \-> UNKNOWN  (timeout; resolve later)

• Transitions use conditional updates: UPDATE payments SET status='SUCCEEDED', version=version+1 WHERE payment_id=? AND status='PROCESSING' AND version=?. Late or duplicate webhooks cannot move a payment backwards.
• The dangerous case is a timeout: we do not know whether the card was charged. Never mark it failed and never blindly charge again. Mark it UNKNOWN, then a background job queries the PSP by reference (or waits for the webhook) and resolves it.
• Persist the intent before calling the PSP. If we crash mid-call, a recovery job finds PROCESSING rows older than N minutes and checks their real status with the PSP.

Double-entry ledger

Every money movement is recorded as a transaction of entries whose debits equal credits. For a $50 order with a $5 platform fee:

• The invariant "sum of debits = sum of credits" is checked on every write and can be audited at any time; a bug that creates or destroys money is detectable immediately.
• Entries are immutable. A refund is a new reversing transaction, not an update or delete.
• Account balances are derived by summing entries; for speed, keep a materialized balance table updated in the same transaction, and periodically verify it against the sum.
• Pay-outs debit seller_42_payable and credit a bank_clearing account once the bank transfer is confirmed.

Reliable events: the transactional outbox

Updating the payments DB and publishing to Kafka are two separate systems; doing both directly risks one succeeding and the other not (dual-write problem). Instead:

1. In the same local transaction as the status change, insert an outbox row.
2. A relay (polling or change-data-capture such as Debezium) publishes outbox rows to Kafka and marks them sent.
3. If the relay crashes, it republishes; consumers deduplicate by event ID.

This gives atomicity between "state changed" and "event will be published" without distributed transactions (2PC). Multi-step flows such as order → payment → inventory use a saga with compensating actions (e.g. refund if inventory reservation fails).

Reconciliation

Even a perfect design drifts: PSP webhooks get lost, banks reverse transfers, currency rounding differs, and humans make manual adjustments. Reconciliation is the safety net.

• Each night, download the PSP's settlement file and compare record by record with our ledger by psp_reference, amount and currency.
• Classify mismatches: missing on our side (lost webhook, fix by replaying), missing on theirs (charge never happened), amount mismatch (fees, FX), or timing differences (settled next day).
• Auto-fix known categories; queue the rest for finance operations with full context.
• Track the mismatch rate as a key health metric and alert on spikes.

Scaling & Reliability

• Stateless services, horizontally scaled; the database is the source of truth.
• Shard by account or merchant if a single primary becomes a bottleneck; keep each ledger transaction within one shard where possible, or use a two-phase "pending then post" pattern across shards.
• Multiple PSPs with smart routing: fail over to a secondary provider when one degrades, and route by card type or region for cost and approval rates.
• Circuit breakers and timeouts around PSP calls; retries with exponential backoff and jitter, always reusing the same idempotency key.
• Synchronous replication for the payments and ledger databases (RPO = 0); losing a committed charge record is unacceptable.
• Hot accounts (the platform fee account receives a credit on every payment) become lock contention points; split into sub-accounts or batch postings.
• Audit and security: encrypt at rest, tokenize payment methods, restrict and log all access, and keep immutable audit trails.

Trade-offs

Interview Tips

• Lead with correctness goals and state that throughput is modest.
• Explain idempotency at every hop: client, PSP call, and event consumers.
• Draw the state machine and dwell on the timeout/unknown case.
• Show a balanced ledger transaction and explain immutability with reversing entries.
• Name the dual-write problem and solve it with the transactional outbox.
• Finish with reconciliation; it shows you know real systems drift.
• Use integer minor units for money and mention currency handling.

References

• System Design Primer: availability vs consistency and asynchronism
• Stripe Engineering: Designing robust and predictable APIs with idempotency
• Stripe API docs: Idempotent requests
• Modern Treasury: Accounting for developers (double-entry)
• ByteByteGo System Design 101