Instagram, Paytm, WhatsApp — different products, same bones. Sketch this in ~30 seconds, then delete what you don't need and thicken what the problem demands.
Client
→ CDN / Edge (static assets, TLS, basic DDoS)
→ Load Balancer (spread traffic)
→ API Gateway (auth, rate limit, routing)
→ App Services (business logic)
→ Cache (hot reads — Redis)
→ Database (source of truth)
→ Blob Store (images, files — S3)
→ Search Index (text queries — Elasticsearch)
→ Queue + Workers (async / slow jobs)
→ Observability (logs, metrics, traces — spans everything)A read = someone views data (scroll feed). A write = someone changes data (post, pay, order).
Reads scale by COPYING — read replicas + cache (100 students photocopy one notebook).
Writes scale by DIVIDING — sharding (split notebook into sections A–H, I–P, Q–Z).
When copies lag, is stale data acceptable for this feature?
Likes / feeds — 2s stale count is fine → eventual consistency, huge scale.
Money / inventory — stale balance or double-booked seat is a lie → strong consistency, slower, worth it.
| Archetype | Spot it by | DB choice | Grow BIG | Keep thin | Signature pattern |
|---|---|---|---|---|---|
| Read-heavy feed | 1000s of reads per write; staleness OK | SQL + replicas + Redis | CDN, cache, fan-out | Strong consistency on hot path | Fan-out-on-write |
| Transactional core | Money, balances, seats — can't double-book | ACID SQL + ledger | Correctness machinery | Cache, CDN on money path | Idempotency key |
| Real-time pipe | Server must push first; ms latency | Cassandra + Redis | WebSockets, pub/sub | Heavy search, complex joins | Connection registry |
| Marketplace | Many entities; no single R:W answer | One specialist DB per service | Depends per counter | Depends per counter | Saga + events |
| Compute pipeline | Answers needed later, not at request time | Lake + columnar warehouse | Queue, batch/stream | Request-time compute | Precompute-and-serve |
Strategy: answer reads without touching the DB — push precomputed feeds as close to the user as possible.
READ: Client → CDN (media) → LB → Feed Service → Redis (precomputed feed) → miss? read replica WRITE: Post Service → Primary DB → Queue → Fan-out workers → insert into each follower's cached feed
- DB: SQL primary + read replicas; denormalized feed lists in Redis — storage cheap, read latency is not
- Name-drop: cache-aside, read replicas, fan-out-on-write, CDN, denormalization
- Trap: celebrity with 100M followers → hybrid fan-out (normal users write-time; celebrities read-time merge)
- Trap: cache invalidation → TTL + accept N seconds staleness, or event-driven eviction
Strategy: one source of truth, append-only facts, every request safe to retry. Sacrifice speed for truth.
Client → Gateway → Payment Service (idempotency key on every request)
↓
ACID SQL — LEDGER (append-only: debit A, credit B; balance = SUM)
↓
outbox table (same TX) → Queue → notifications / analytics- DB: Postgres/MySQL — never mutate balances; append immutable entries
- Name-drop: idempotency keys, double-entry ledger, outbox pattern, saga, pessimistic locking for seats
- Trap: payment spans two services → saga with compensating actions
- Trap: "cache the balance?" → stale balance is wrong data, not slow data
Strategy: keep a permanent open line (WebSocket) per user; route between gateway servers via registry + pub/sub.
Phone A ══ WebSocket ══ Gateway 1 ──► Connection Registry (Redis: user B → server 2)
│
Phone B ══ WebSocket ══ Gateway 2 ◄── Pub/Sub bus
│
Cassandra (history) + Queue (offline push)- DB: Cassandra partitioned by conversation_id; Redis for presence, typing, unread
- Name-drop: WebSockets, pub/sub, connection registry, heartbeats, per-conversation sequence numbers
- Trap: gateway crash → clients reconnect with backoff; pull missed msgs from history
- Trap: 500-person group chat = fan-out at the socket layer (same idea as feeds)
Strategy: decompose into sub-systems; classify each with Q1/Q2; one specialist DB per service; connect with events.
Gateway ├─ Catalog (document DB + cache) read-heavy ├─ Search (Elasticsearch) librarian ├─ Cart (Redis) ephemeral ├─ Orders + Inventory (SQL + locks) can't oversell └─ Payments (Archetype 2 ledger) money path Checkout saga via Queue: order → pay → reserve stock → assign delivery (each step has undo)
- Name-drop: database-per-service, saga, CQRS, optimistic vs pessimistic inventory locks
- Trap: flash sale 10K buyers / 100 units → atomic
UPDATE … WHERE qty > 0is the choke point; queue the rest
Strategy: queue absorbs the flood; store raw cheap; batch/stream process; precompute so the dashboard read is instant.
Events → Kafka (buffer, nothing dropped)
├─ Stream (Flink) — near-live approximate counters
└─ Batch (Spark) — nightly exact aggregates
↓
Data Lake (S3) → Columnar warehouse (BigQuery / ClickHouse)
↓
Serving cache → dashboard (never compute at request time)- Name-drop: queue-buffered ingestion, ETL, precompute-and-serve, idempotent replayable consumers
- Trap: "need near-real-time?" → stream path for fresh-approximate + batch for exact (Lambda architecture)
2. Q1 — add expense vs check balance? Both tiny; scale is NOT the hard problem → skip CDN, skip sharding; modest cache only.
3. Q2 — it's money → Archetype 2: Postgres, append-only expense ledger, idempotency keys, derive balances from facts.
4. Spend remaining minutes on the real hard problem: debt-simplification graph (A owes B, B owes C → minimize transfers).
| Database | Type | Consistency | Query style | Best for | Avoid when |
|---|---|---|---|---|---|
| PostgreSQL / MySQL | Relational | Strong (ACID) | SQL — joins, aggregations, transactions | Payments, orders, user accounts, anything needing joins + ACID | Write throughput > 50K/s; massive scale-out; schema-less data |
| MongoDB | Document | Strong (configurable) | JSON-like queries, flexible schema | Catalog data, user profiles, content with variable fields | Complex joins, strict transactions across many docs, financial data |
| Cassandra / DynamoDB | Wide-column | Tunable (eventual → strong) | Partition key + range queries; no joins | Messaging, activity feeds, time-series at massive write scale | Ad-hoc queries, complex aggregations, strong consistency everywhere |
| Redis | In-memory KV | Eventual (not durable by default) | Key lookup, sorted sets, pub/sub, Lua scripts | Cache, session store, leaderboards, rate limiting, ephemeral state | Durable source of truth; data larger than RAM; complex queries |
| Elasticsearch | Search / inverted index | Eventually consistent | Full-text search, geo queries, aggregations | Search boxes, log analytics, geo-filtered queries | Primary data store; financial transactions; exact consistency |
| InfluxDB / TimescaleDB | Time-series | Strong (time-ordered) | Time-range aggregations; append-only writes | Metrics, monitoring, IoT, anything queried by time range | Random access by non-time keys; transactional workloads |
| Neo4j | Graph | Strong (ACID) | Cypher — graph traversals, shortest path | Social graphs, fraud detection, recommendation engines | Large-scale writes; non-graph data; most interview scenarios |
- Reach for it when: you need ACID, joins, foreign key constraints, or complex aggregations
- Scale ceiling: ~50K writes/s on a single node; vertical scale first, then read replicas
- Scale-out: sharding by partition key (user_id, tenant_id); horizontal sharding adds complexity
- Indexing: B-tree indexes on PK + foreign keys; partial indexes for filtered queries
- Tradeoff: harder to scale writes horizontally vs NoSQL; schema migrations at large scale are painful
- Reach for it when: data is JSON-shaped, schema varies per document, or rapid iteration matters
- Scale: horizontal sharding built-in; replica sets for HA
- Consistency: strong at document level; multi-document transactions available but expensive
- Watch out for: denormalization — embedded vs referenced documents is a key schema decision
- Tradeoff: flexible reads but multi-collection joins are expensive — denormalize deliberately
- Reach for it when: write throughput is massive, reads are simple key lookups, or data is append-only
- Data model: design around your access patterns first — partition key determines node, clustering key determines order
- Consistency: tunable — QUORUM for strong, ONE for speed; eventual by default
- No joins: denormalize everything; one table per query pattern
- Tradeoff: exceptional write scale but inflexible for ad-hoc queries; schema changes are operational work
- Reach for it when: latency must be sub-millisecond, or data is ephemeral / frequently accessed
- Data structures: strings, hashes, sorted sets (leaderboard), lists, pub/sub, streams
- Durability: RDB snapshots + AOF log — but never use Redis as your only source of truth for critical data
- Lua scripts: atomic multi-step operations (rate limiting, CAS patterns)
- Tradeoff: data must fit in RAM; not durable by default; cluster adds complexity
- Reach for it when: query requires full-text search, fuzzy match, faceting, or geo_distance filters
- Sync pattern: never make ES your source of truth — sync from your primary DB via CDC or write-through
- Sharding: documents sharded by hash; each shard is a Lucene index
- Lag: index updates are near-real-time (~1s) but not instant — acceptable for most search use cases
- Tradeoff: operational complexity; sync lag; not suitable for ACID or primary storage
- Reach for it when: data is append-only time-stamped measurements queried by time range
- Rollups: auto-downsample older data — raw 24h → 1min 30d → 1hr forever
- Cardinality: each unique metric + tag combination = a new series; high-cardinality tags destroy performance
- Partitioning: time-based partitioning means old data drops off cheaply (TTL)
- Tradeoff: purpose-built = fast for time-range queries; inflexible for non-time workloads
- Reach for it when: the relationships between entities are the primary query dimension
- Query language: Cypher — intuitive for path traversals, shortest paths, recommendations
- Strength: k-hop queries that would require k JOINs in SQL are O(1) per hop in a graph DB
- Scale ceiling: works well up to billions of nodes but write scale is lower than wide-column DBs
- Tradeoff: rarely the right answer in interviews unless the problem is explicitly social graph, fraud, or recommendations
- Reach for it when: storing large binary objects — images, videos, audio, backups, CSVs
- Rule: never store blobs in a relational DB — use object storage + store only the URL in the DB
- Access pattern: immutable objects with a URL; versioning support; lifecycle rules for TTL
- CDN pairing: always put a CDN in front of S3 for read-heavy access
- Cost: cheapest storage per GB — orders of magnitude cheaper than DB storage
- Using Redis as primary storage for anything that must survive a restart. Redis is a cache first. Always have a durable source of truth behind it.
- Storing BLOBs in PostgreSQL — images, videos, large files belong in S3 + CDN. Only store the object URL in the DB.
- Using Cassandra for read-heavy, ad-hoc queries — Cassandra is optimized for known access patterns. If you need flexibility, use Postgres or Elasticsearch.
- Making Elasticsearch your source of truth — ES index is a derived view. Always sync from a primary DB. An ES failure should not lose data.
- Choosing a DB before knowing your access patterns — always derive the data model from your API design. The access pattern determines the schema.
- Using a single DB for everything without justification — but also don't add DBs needlessly. Polyglot persistence adds operational cost. Only specialize when you have a clear reason.
- Forgetting cardinality in time-series — using a high-cardinality label (like user_id) as a tag in InfluxDB creates billions of series and destroys performance.
- Skipping the "why" when naming a DB — saying "I'd use Cassandra" without explaining why is a missed signal. Always say: "I'd use Cassandra because writes are append-only at massive scale and I only need partition key lookups."
| System | CAP | Interview line | Real-world use |
|---|---|---|---|
| PostgreSQL / MySQL | CP | Strong ACID; blocks or fails on partition rather than serve stale writes | Payments, orders, seat inventory, ledgers |
| MongoDB | CP | Primary-only writes; read from secondaries may be stale unless you force primary | User profiles, product catalogs, config stores |
| Cassandra / DynamoDB | AP (tunable) | Available during partition; QUORUM gives CP-like behavior at cost of latency | Messaging, timelines, activity logs, IoT events |
| Redis | AP | Fast and available; never the sole source of truth for critical durable data | Cache, sessions, rate limits, leaderboards |
| Elasticsearch | AP | Eventually consistent index; rebuild from primary DB if index is lost | Search, log analytics, geo-filtered queries |
| ZooKeeper / etcd | CP | Coordination and leader election; blocks writes during partition | Kafka broker coordination, distributed locks, service discovery |
| Google Spanner | CP + external | Global strong consistency via TrueTime — rare, expensive, not a default answer | Global billing, cross-region financial ledgers |
70+ patterns with staff answer + trade-offs + examples — interview-quick-fire.html (best on GitHub). Includes classic failure modes (thundering herd, retry storm, hot key, split brain). Below: top 10 to drill before a mock.
SELECT FOR UPDATE (pessimistic) or version-field CAS (optimistic). Idempotency key on every write. Stripe deduplicates retries via idempotency keys.+60 more: thundering herd, metastable failure, poison messages, sagas, rate limits, payments → interview-quick-fire.html
- Use when: money moves, inventory decrements, anything that can't be wrong once
- How: single DB transaction,
SELECT FOR UPDATE, or distributed 2PC (last resort) - Tradeoff: limits throughput; cross-service transactions are slow and fragile
- Use when: feeds, analytics, search indexes, non-critical reads
- How: write to primary, replicate async; readers may see stale data for seconds
- Tradeoff: simpler and faster; must handle stale reads in UI or rebuild on failure
- Use when: message queues, payment webhooks, job retries
- How: idempotency key on every request; consumer checks "already processed?" before acting
- Tradeoff: requires dedup store (Redis/DB); duplicates are possible but harmless
- Use when: read and write patterns differ radically (feeds, search, analytics)
- How: write to OLTP store; async workers build optimized read views (ES, Redis, Cassandra)
- Tradeoff: sync lag; more moving parts; read model can be rebuilt from write log
Full tables with CLI commands live in v10 appendix. In interviews, map patterns to managed services:
aws s3 cp file.mp4 s3://bucket/key — multipart for large filesaws s3 presign s3://bucket/key --expires-in 3600 — temporary client upload URLaws sqs send-message --queue-url URL --message-body '{}'aws dynamodb put-item --table-name T --item '{"pk":{"S":"x"}}'redis-cli SET key val EX 3600 — TTL cache entryaws kinesis put-record --stream-name s --partition-key k --data $(echo '{}'|base64)Bitly solves short link creation and fast redirection at scale. You take a long URL, store a mapping to a shorter code, and when someone visits the short link the system looks up the original and redirects them.
The real challenges: making short codes unique, keeping redirects fast, and handling a read-heavy workload — links are clicked far more than they are created.
The hard part is that reads explode while writes stay small. A Bitly style system might create URLs slowly, but a single popular link can suddenly cause huge redirect traffic, so you need very fast lookups and a way to survive spikes.
The other tricky part is global uniqueness for short codes. If you scale the write path across many servers, they all need to agree on which code comes next or you risk collisions. So the two main scaling pain points are read traffic on redirects and coordination for code generation.
- Scope. URL in, short URL out. Short URL in, redirect out. Keep analytics and auth out of scope unless asked.
- Key requirement. Read-heavy system — optimize redirects first.
- API. POST /urls creates a short link. GET /{shortCode} redirects.
- Data model. short_code → long_url + created time + optional expiration. short_code is the primary key.
- Code generation. Global counter + base62 is the best default. Easy to explain, guarantees uniqueness.
- Fast reads. Redis cache in front of DB. Cache first, DB on miss.
- Redirect. Return 302 not 301 — keeps control and avoids permanent browser caching.
- Expiration. Check expiry on read. Return 410 Gone if expired. Match cache TTL to expiration.
- Scale. Stateless app servers behind a load balancer. Split read and write services if needed.
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
The scaling pain here is coordination — every server needs a unique code without collision
Describe hashing and move on
articulate a progression of approaches and their tradeoffs. Start with MD5/SHA-256 + Base62 truncation: simple but has collision probability that grows as n/|S|. The fix is a uniqueness check + retry, which adds a DB roundtrip. The better default is a global atomic counter in Redis (INCR is single-threaded, atomic, eliminates collisions entirely) with Base62 encoding. The staff-level concern is the counter as a single point of failure: pre-allocated ID ranges per app server (each batch-fetches 1,000 IDs, eliminating per-request Redis coordination), and the counter node failing means temporary unavailability of writes but no data loss
Mention the predictability risk: sequential codes are enumerable. Mitigation: XOR the counter with a secret key before encoding, or accept that short URLs are meant to be shared publicly anyway
add a database index on short_code and query on every redirect
this is what makes Bitly hard — 100M DAU clicking links 10× per day = 11,600 read QPS, with viral links creating hot keys at 100× that rate. A database index on short_code is necessary but not sufficient. The right answer layers caching: Redis with cache-aside pattern absorbs 99% of reads
cache TTL should be set equal to or shorter than URL expiration so expired URLs auto-evict from cache (otherwise you serve expired redirects). For hot keys (viral links), add a local in-process LRU cache on each app server — zero network hops, handles traffic spikes that would otherwise hammer Redis. For global-scale: CDN caching of the 302 response itself with a short Cache-Control header removes Redis from the critical path entirely for the most popular links
scale the database vertically and add read replicas
storage is not the hard part — 1B × 500 bytes = 500 GB, comfortably on a single PostgreSQL node with read replicas. The hard part is write coordination for the counter and horizontal scaling of stateless redirect servers. Senior answer: stateless redirect service behind a load balancer, Redis cluster for the counter, PG primary + read replicas
the counter service is the hidden bottleneck — pre-allocated ID ranges mean app servers can generate IDs locally without any network call (counter node becomes a periodic flush rather than a per-request bottleneck). Multi-region: deploy read replicas and Redis caches regionally, write counter remains in one region (acceptable because writes are rare). For URL expiration at scale: don't run a full table scan — use a time-indexed expiry column and batch-delete expired rows during low-traffic windows
Why the deep dives connect to the scaling problem: The scaling pain is "reads explode while writes stay small." Deep dive 1 solves write uniqueness. Deep dive 2 solves read performance. Deep dive 3 solves infrastructure capacity. Name this arc explicitly in the interview — it shows architectural thinking, not just pattern recall.
+-------------------+
| Clients |
| browser mobile app|
+---------+---------+
|
v
+-------------------+
| Load Balancer |
+----+---------+----+
| |
write path | | read path
v v
+----------------+ +----------------+
| Write Service | | Read Service |
+-------+--------+ +-------+--------+
| |
| |
v v
+----------------+ +----------------+
| Redis Counter | | Redis Cache |
| atomic INCR | | short -> long |
+-------+--------+ +-------+--------+
| |
| | cache miss
| v
| +--------------------+
+---------->| Postgres |
| short_code PK |
| long_url |
| expiration |
+---------+----------+
|
v
+--------------------+
| Background Cleanup |
| delete expired URLs|
+--------------------+If you want to say it out loud, keep it simple. Clients hit a load balancer. Writes go to a write service, which gets a unique ID from Redis Counter, converts it to base62, and stores the mapping in Postgres. Reads go to a read service, which checks Redis Cache first, falls back to Postgres on a miss, checks expiration, and returns a 302 redirect.
If you want a slightly stronger version, you could add a CDN in front of the read path for hot links, but I would start with the sketch above in an interview.
Problem
Bitly solves short link creation and fast redirection at scale. You take a long URL, store a mapping to a shorter code, and when someone visits the short link the system looks up the original and redirects them.
The real challenges: making short codes unique, keeping redirects fast, and handling a read-heavy workload — links are clicked far more than they are created.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is that reads explode while writes stay small. A Bitly style system might create URLs slowly, but a single popular link can suddenly cause huge redirect traffic, so you need very fast lookups and a way to survive spikes.
The other tricky part is global uniqueness for short codes. If you scale the write path across many servers, they all need to agree on which code comes next or you risk collisions. So the two main scaling pain points are read traffic on redirects and coordination for code generation.
Key points
- Scope. URL in, short URL out. Short URL in, redirect out. Keep analytics and auth out of scope unless asked.
- Key requirement. Read-heavy system — optimize redirects first.
- API. POST /urls creates a short link. GET /{shortCode} redirects.
- Data model. short_code → long_url + created time + optional expiration. short_code is the primary key.
- Code generation. Global counter + base62 is the best default. Easy to explain, guarantees uniqueness.
- Fast reads. Redis cache in front of DB. Cache first, DB on miss.
- Redirect. Return 302 not 301 — keeps control and avoids permanent browser caching.
- Expiration. Check expiry on read. Return 410 Gone if expired. Match cache TTL to expiration.
- Scale. Stateless app servers behind a load balancer. Split read and write services if needed.
Tradeoffs
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
The scaling pain here is coordination — every server needs a unique code without collision
Describe hashing and move on
articulate a progression of approaches and their tradeoffs. Start with MD5/SHA-256 + Base62 truncation: simple but has collision probability that grows as n/|S|. The fix is a uniqueness check + retry, which adds a DB roundtrip. The better default is a global atomic counter in Redis (INCR is single-threaded, atomic, eliminates collisions entirely) with Base62 encoding. The staff-level concern is the counter as a single point of failure: pre-allocated ID ranges per app server (each batch-fetches 1,000 IDs, eliminating per-request Redis coordination), and the counter node failing means temporary unavailability of writes but no data loss
Mention the predictability risk: sequential codes are enumerable. Mitigation: XOR the counter with a secret key before encoding, or accept that short URLs are meant to be shared publicly anyway
add a database index on short_code and query on every redirect
this is what makes Bitly hard — 100M DAU clicking links 10× per day = 11,600 read QPS, with viral links creating hot keys at 100× that rate. A database index on short_code is necessary but not sufficient. The right answer layers caching: Redis with cache-aside pattern absorbs 99% of reads
cache TTL should be set equal to or shorter than URL expiration so expired URLs auto-evict from cache (otherwise you serve expired redirects). For hot keys (viral links), add a local in-process LRU cache on each app server — zero network hops, handles traffic spikes that would otherwise hammer Redis. For global-scale: CDN caching of the 302 response itself with a short Cache-Control header removes Redis from the critical path entirely for the most popular links
scale the database vertically and add read replicas
storage is not the hard part — 1B × 500 bytes = 500 GB, comfortably on a single PostgreSQL node with read replicas. The hard part is write coordination for the counter and horizontal scaling of stateless redirect servers. Senior answer: stateless redirect service behind a load balancer, Redis cluster for the counter, PG primary + read replicas
the counter service is the hidden bottleneck — pre-allocated ID ranges mean app servers can generate IDs locally without any network call (counter node becomes a periodic flush rather than a per-request bottleneck). Multi-region: deploy read replicas and Redis caches regionally, write counter remains in one region (acceptable because writes are rare). For URL expiration at scale: don't run a full table scan — use a time-indexed expiry column and batch-delete expired rows during low-traffic windows
Why the deep dives connect to the scaling problem: The scaling pain is "reads explode while writes stay small." Deep dive 1 solves write uniqueness. Deep dive 2 solves read performance. Deep dive 3 solves infrastructure capacity. Name this arc explicitly in the interview — it shows architectural thinking, not just pattern recall.
Interview script
Whiteboard
+-------------------+
| Clients |
| browser mobile app|
+---------+---------+
|
v
+-------------------+
| Load Balancer |
+----+---------+----+
| |
write path | | read path
v v
+----------------+ +----------------+
| Write Service | | Read Service |
+-------+--------+ +-------+--------+
| |
| |
v v
+----------------+ +----------------+
| Redis Counter | | Redis Cache |
| atomic INCR | | short -> long |
+-------+--------+ +-------+--------+
| |
| | cache miss
| v
| +--------------------+
+---------->| Postgres |
| short_code PK |
| long_url |
| expiration |
+---------+----------+
|
v
+--------------------+
| Background Cleanup |
| delete expired URLs|
+--------------------+If you want to say it out loud, keep it simple. Clients hit a load balancer. Writes go to a write service, which gets a unique ID from Redis Counter, converts it to base62, and stores the mapping in Postgres. Reads go to a read service, which checks Redis Cache first, falls back to Postgres on a miss, checks expiration, and returns a 302 redirect.
If you want a slightly stronger version, you could add a CDN in front of the read path for hot links, but I would start with the sketch above in an interview.
Dropbox solves cloud file storage and sync. Let users upload a file once, store it reliably, access it from any device, share it with others, and keep copies in sync.
Two layers: durable blob storage for the actual file bytes, and metadata + change tracking for ownership, sharing, and sync.
The hard part is moving and syncing very large files cheaply and reliably. In Dropbox, the bottleneck is not just request count. It is the sheer amount of data, long upload times, and the fact that users expect the same file to appear quickly across devices.
A few things make this hard. Large files can time out, fail halfway, and need resume support. Downloads are heavy too, especially for users far from your storage region, so you need direct blob storage access and often a CDN. Sync is also tricky because you need to notice file changes, push updates to other devices, and recover if a device misses a notification. On top of that, sharing and permissions add metadata lookups, while reliability means you need durable storage and recovery if a server dies. So the core scaling pain is big blobs plus cross-device coordination, not just more API servers.
- Metadata through your service. Control plane: auth, metadata, sharing, URL signing. Heavy file bytes go around your service, not through it.
- Blob storage. Store actual file contents in S3. Never proxy large files through app servers.
- Pre-signed URL upload. Client uploads directly to blob storage. Bypasses your servers entirely.
- CDN download. Signed CDN URL for downloads. Low latency for users far away.
- Chunking. Split large files into 4 MB chunks on the client. Gives progress, retry, and resume.
- SHA-256 dedup. Same hash = same content = already stored. Global dedup across all users for free.
- Delta sync. On update, only upload changed chunks. This is what makes sync feel fast on large files.
- Sharing. Separate SharedFiles table — makes "files shared with me" queries fast.
The scaling pain is moving and syncing large files cheaply and reliably
upload file to S3, done
client-side chunking (4 MB chunks) with SHA-256 fingerprint per chunk. Before uploading, client sends hash list to server — server responds with which hashes are new. Only missing chunks are transferred. This is content-addressable storage: same chunk stored once globally
chunk size is a design decision with real tradeoffs — smaller chunks give better dedup granularity and retry precision but more metadata overhead and round-trips; larger chunks have less overhead but worse dedup and waste bandwidth if a chunk fails mid-upload. Content-defined chunking (CDC using rolling hash, e.g., Rabin fingerprint) gives better dedup ratios for structured files (docs, code) by finding natural chunk boundaries rather than fixed offsets. The upload flow must be resumable: store upload state per (file_id, chunk_offset) so a failed upload resumes from the last committed chunk, not from scratch
The scaling pain is cross-device coordination — detecting changes, pushing updates, and recovering from missed notifications
polling
WebSocket or SSE push for change notifications, with polling as a fallback for reconnect. The hard problem is what happens when two devices edit the same file while one is offline. Dropbox's actual approach: last-write-wins per file, conflicts result in a conflict copy (two files) rather than silent data loss
vector clocks or file version numbers allow the server to detect when a client's local state diverges from the server's state, enabling explicit conflict presentation rather than silent overwrites. The sync state machine per device: (synced → local_change_pending → uploading → synced) and (synced → remote_change_available → downloading → synced). On reconnect, client sends its last-known state vector; server diffs and returns the minimal change set
store the file URL in a shared link, serve directly from S3 with a public ACL
sharing model: SharedFiles table (file_id, shared_with_user_id, permission, created_at)
permission check on every file operation must be fast — cache permission results in Redis with short TTL, invalidate on any permission change. For large organizations: hierarchical permissions (team → folder → file) require careful data modeling to avoid O(n) permission lookups. Security: pre-signed S3 URLs with short expiry (15 min) for downloads — URL cannot be reused after expiry, prevents link sharing beyond the intended recipient. For compliance: server-side encryption (SSE-S3 or SSE-KMS), access logs for auditing, retention policies. Performance: CDN in front of S3 with signed cookies for authenticated users — reduces S3 egress costs and improves global latency
Why the deep dives connect to the scaling problem: "Big blobs plus cross-device coordination." Deep dive 1 solves the blob problem (chunking + dedup). Deep dive 2 solves coordination (sync state machine + conflict handling). Deep dive 3 solves correctness and performance at scale.
Use a simple script: upload, download, sharing, sync, then one deep dive.
+----------------------+
| Desktop / Mobile |
| Web Client + Sync |
| Agent |
+----------+-----------+
|
auth, metadata APIs, change feed
|
v
+----------------------+
| LB / API Gateway |
+----------+-----------+
|
v
+----------------------+
| File Service |
| - authz checks |
| - file metadata |
| - share management |
| - presigned URLs |
| - signed CDN URLs |
+----+------------+----+
| |
metadata rw | | change events
v v
+----------------+ +------------------+
| FileMetadataDB | | Notification / |
| - files | | Change Service |
| - sharedFiles | | - WebSocket/SSE |
| - upload state | +--------+---------+
+----------------+ |
|
push updates |
v
+---------------+
| Client devices|
+---------------+
Upload path
-----------
Client -> File Service -> get presigned upload URL
Client -------------------------------> Blob Storage / S3
|
| upload complete event
v
File Service updates DB
Download path
-------------
Client -> File Service -> auth check + signed CDN URL
Client -------------------------------> CDN
|
cache miss|
v
Blob Storage / S3
Sharing path
------------
Client -> File Service -> update share records in DB
Sync path
---------
Local file change -> Sync Agent -> upload flow
Remote file change -> Notification Service pushes event
Missed event fallback -> Client polls `GET /files/changes?since=...`The main mental model is this. Your app server is the control plane, not the data plane. It decides who can access a file and issues signed URLs, but the heavy file bytes go directly between the client, blob storage, and CDN.
If you want, I can also give you a more interview ready version that is smaller and faster to draw on a whiteboard.
Problem
Dropbox solves cloud file storage and sync. Let users upload a file once, store it reliably, access it from any device, share it with others, and keep copies in sync.
Two layers: durable blob storage for the actual file bytes, and metadata + change tracking for ownership, sharing, and sync.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is moving and syncing very large files cheaply and reliably. In Dropbox, the bottleneck is not just request count. It is the sheer amount of data, long upload times, and the fact that users expect the same file to appear quickly across devices.
A few things make this hard. Large files can time out, fail halfway, and need resume support. Downloads are heavy too, especially for users far from your storage region, so you need direct blob storage access and often a CDN. Sync is also tricky because you need to notice file changes, push updates to other devices, and recover if a device misses a notification. On top of that, sharing and permissions add metadata lookups, while reliability means you need durable storage and recovery if a server dies. So the core scaling pain is big blobs plus cross-device coordination, not just more API servers.
Key points
- Metadata through your service. Control plane: auth, metadata, sharing, URL signing. Heavy file bytes go around your service, not through it.
- Blob storage. Store actual file contents in S3. Never proxy large files through app servers.
- Pre-signed URL upload. Client uploads directly to blob storage. Bypasses your servers entirely.
- CDN download. Signed CDN URL for downloads. Low latency for users far away.
- Chunking. Split large files into 4 MB chunks on the client. Gives progress, retry, and resume.
- SHA-256 dedup. Same hash = same content = already stored. Global dedup across all users for free.
- Delta sync. On update, only upload changed chunks. This is what makes sync feel fast on large files.
- Sharing. Separate SharedFiles table — makes "files shared with me" queries fast.
Tradeoffs
Deep dives
The scaling pain is moving and syncing large files cheaply and reliably
upload file to S3, done
client-side chunking (4 MB chunks) with SHA-256 fingerprint per chunk. Before uploading, client sends hash list to server — server responds with which hashes are new. Only missing chunks are transferred. This is content-addressable storage: same chunk stored once globally
chunk size is a design decision with real tradeoffs — smaller chunks give better dedup granularity and retry precision but more metadata overhead and round-trips; larger chunks have less overhead but worse dedup and waste bandwidth if a chunk fails mid-upload. Content-defined chunking (CDC using rolling hash, e.g., Rabin fingerprint) gives better dedup ratios for structured files (docs, code) by finding natural chunk boundaries rather than fixed offsets. The upload flow must be resumable: store upload state per (file_id, chunk_offset) so a failed upload resumes from the last committed chunk, not from scratch
The scaling pain is cross-device coordination — detecting changes, pushing updates, and recovering from missed notifications
polling
WebSocket or SSE push for change notifications, with polling as a fallback for reconnect. The hard problem is what happens when two devices edit the same file while one is offline. Dropbox's actual approach: last-write-wins per file, conflicts result in a conflict copy (two files) rather than silent data loss
vector clocks or file version numbers allow the server to detect when a client's local state diverges from the server's state, enabling explicit conflict presentation rather than silent overwrites. The sync state machine per device: (synced → local_change_pending → uploading → synced) and (synced → remote_change_available → downloading → synced). On reconnect, client sends its last-known state vector; server diffs and returns the minimal change set
store the file URL in a shared link, serve directly from S3 with a public ACL
sharing model: SharedFiles table (file_id, shared_with_user_id, permission, created_at)
permission check on every file operation must be fast — cache permission results in Redis with short TTL, invalidate on any permission change. For large organizations: hierarchical permissions (team → folder → file) require careful data modeling to avoid O(n) permission lookups. Security: pre-signed S3 URLs with short expiry (15 min) for downloads — URL cannot be reused after expiry, prevents link sharing beyond the intended recipient. For compliance: server-side encryption (SSE-S3 or SSE-KMS), access logs for auditing, retention policies. Performance: CDN in front of S3 with signed cookies for authenticated users — reduces S3 egress costs and improves global latency
Why the deep dives connect to the scaling problem: "Big blobs plus cross-device coordination." Deep dive 1 solves the blob problem (chunking + dedup). Deep dive 2 solves coordination (sync state machine + conflict handling). Deep dive 3 solves correctness and performance at scale.
Interview script
Use a simple script: upload, download, sharing, sync, then one deep dive.
Whiteboard
+----------------------+
| Desktop / Mobile |
| Web Client + Sync |
| Agent |
+----------+-----------+
|
auth, metadata APIs, change feed
|
v
+----------------------+
| LB / API Gateway |
+----------+-----------+
|
v
+----------------------+
| File Service |
| - authz checks |
| - file metadata |
| - share management |
| - presigned URLs |
| - signed CDN URLs |
+----+------------+----+
| |
metadata rw | | change events
v v
+----------------+ +------------------+
| FileMetadataDB | | Notification / |
| - files | | Change Service |
| - sharedFiles | | - WebSocket/SSE |
| - upload state | +--------+---------+
+----------------+ |
|
push updates |
v
+---------------+
| Client devices|
+---------------+
Upload path
-----------
Client -> File Service -> get presigned upload URL
Client -------------------------------> Blob Storage / S3
|
| upload complete event
v
File Service updates DB
Download path
-------------
Client -> File Service -> auth check + signed CDN URL
Client -------------------------------> CDN
|
cache miss|
v
Blob Storage / S3
Sharing path
------------
Client -> File Service -> update share records in DB
Sync path
---------
Local file change -> Sync Agent -> upload flow
Remote file change -> Notification Service pushes event
Missed event fallback -> Client polls `GET /files/changes?since=...`The main mental model is this. Your app server is the control plane, not the data plane. It decides who can access a file and issues signed URLs, but the heavy file bytes go directly between the client, blob storage, and CDN.
If you want, I can also give you a more interview ready version that is smaller and faster to draw on a whiteboard.
PostGIS / geohash
decrement stock
Given a user's location, show which items are available for delivery within 1 hour, and let the user place an order without selling the same physical inventory twice.
The hard part: fast reads for availability, strongly consistent writes for orders.
The hard part is that reads are huge, but writes must be correct. In this Gopuff style system, availability checks happen constantly and need to stay under 100ms, while actual orders are less frequent but need strong consistency so you never sell the same item twice.
So there are really two scaling pain points. First, availability is expensive because you are not just reading one row. You have to find nearby distribution centers, read inventory from several of them, and union the results fast enough for search-like traffic. Second, ordering creates contention because many users may try to buy the last unit at the same time, so inventory updates need atomic transactions or locking. A good mental model is that reads are broad and frequent, while writes are rare but delicate.
- Two-path model. Fast reads for availability. Safe writes for orders. Different consistency requirements.
- Availability read. Find nearby DCs via geohash, aggregate inventory, cache in Redis with short TTL.
- Order write. Recheck inventory and create order in one PostgreSQL serializable transaction.
- Nearby service. Coarse geographic filter first, then travel time estimation on the small candidate set.
- Cache invalidation. Expire affected Redis keys after an order decrements inventory.
query PostgreSQL with PostGIS radius filter on every availability request
the scaling pain is that availability reads are expensive: find nearby DCs, read inventory from each, union results, respond in <100ms. This is not a simple DB read — it's a geospatial + multi-source aggregation under search-like QPS. Weak answer: query PG with PostGIS radius filter. Strong answer: pre-compute a geohash index of all DCs, cache inventory in Redis with short TTL (30s). Availability query = geohash prefix lookup (O(1)) + Redis HGETALL for each DC in range
inventory in Redis is a projection, not the source of truth — it gets stale within the TTL window. This is intentional: users browsing see approximately-fresh inventory, users purchasing get a real-time DB check. The two paths have different consistency requirements and must be explicitly separated. At high QPS, even Redis can become a bottleneck — read-through cache with local in-process LRU for the hottest items (fast movers, popular items) adds another layer
The core hard problem: many users buying the last unit simultaneously
check inventory then decrement (two operations — TOCTOU race)
SELECT FOR UPDATE within a PostgreSQL transaction — atomic lock, check, and decrement
tradeoff: pessimistic locking works but creates serialization under high concurrent orders for the same item. Optimistic locking (CAS via version number) is better under low contention, worse under high contention. For a local delivery system where popular items genuinely spike (weather events, flash sales), pessimistic is the safer default. SQS queue for order submission acts as a shock absorber: workers drain the queue at a safe DB write rate, and user sees "order processing" rather than an error. Idempotency key on every order prevents double-charge on client retry
find the nearest DC by Euclidean distance
a production system doesn't just find the nearest DC by Euclidean distance — it finds the nearest DC that can fulfill the order within the delivery window. This requires: (1) drive time estimate from DC to delivery address (routing API, not just radius), (2) DC capacity check (is there a driver available?), (3) inventory check (does this DC have the items?)
this is a constraint satisfaction problem across three dimensions. The production approach is coarse + fine: geohash radius filter first (fast, eliminates 99% of DCs), then routing API call on the small candidate set (expensive, accurate). Cache routing estimates per (DC_geohash, delivery_zone) pair with 5-minute TTL to reduce routing API costs
Why the deep dives connect to the scaling problem: "Reads are broad and frequent; writes are rare but delicate." Deep dive 1 solves the read problem. Deep dive 2 solves the write correctness problem. Deep dive 3 solves the geo-routing precision problem.
Two-path script.
+------------------+
| Client |
| Web or Mobile |
+---------+--------+
|
Availability API | Order API
|
+----------------+----------------+
| |
v v
+-----------------------+ +-----------------------+
| Availability Service | | Orders Service |
| read path | | write path |
+----------+------------+ +-----------+-----------+
| |
| asks for serviceable DCs | asks for serviceable DCs
v v
+---------------------------+
| Nearby Service |
| find DCs within 1 hour |
+-------------+-------------+
|
| candidate DCs
v
+---------------------------+
| Travel Time Service |
| external ETA estimation |
+---------------------------+
Availability read flow
----------------------
|
v
+-----------------------+
| Redis Cache |
| availability results |
| short TTL |
+----------+------------+
|
cache miss
v
+-----------------------+
| Postgres Read Replica |
| inventory reads |
+----------+------------+
|
v
+-----------------------+
| Partitioned by Region |
| inventory + items |
+-----------------------+
Order write flow
----------------
|
v
+-----------------------+
| Postgres Leader |
| serializable txn |
+----------+------------+
|
v
+-----------------------+
| Tables |
| Inventory |
| Items |
| Orders |
| OrderItems |
+-----------------------+
|
v
+-----------------------+
| Cache Invalidation |
| expire affected keys |
+-----------------------+The mental model is simple. Availability is a fast read path that can tolerate slight staleness, so it uses Nearby Service, cache, and read replicas. Orders are the strict write path, so they go to the Postgres leader in one transaction so you do not double sell inventory.
If you were drawing this in an interview, I would show just these boxes first. Then I would say reads go through cache and replicas, while writes go through the leader with an atomic transaction.
Problem
Given a user's location, show which items are available for delivery within 1 hour, and let the user place an order without selling the same physical inventory twice.
The hard part: fast reads for availability, strongly consistent writes for orders.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is that reads are huge, but writes must be correct. In this Gopuff style system, availability checks happen constantly and need to stay under 100ms, while actual orders are less frequent but need strong consistency so you never sell the same item twice.
So there are really two scaling pain points. First, availability is expensive because you are not just reading one row. You have to find nearby distribution centers, read inventory from several of them, and union the results fast enough for search-like traffic. Second, ordering creates contention because many users may try to buy the last unit at the same time, so inventory updates need atomic transactions or locking. A good mental model is that reads are broad and frequent, while writes are rare but delicate.
Key points
- Two-path model. Fast reads for availability. Safe writes for orders. Different consistency requirements.
- Availability read. Find nearby DCs via geohash, aggregate inventory, cache in Redis with short TTL.
- Order write. Recheck inventory and create order in one PostgreSQL serializable transaction.
- Nearby service. Coarse geographic filter first, then travel time estimation on the small candidate set.
- Cache invalidation. Expire affected Redis keys after an order decrements inventory.
Tradeoffs
Deep dives
query PostgreSQL with PostGIS radius filter on every availability request
the scaling pain is that availability reads are expensive: find nearby DCs, read inventory from each, union results, respond in <100ms. This is not a simple DB read — it's a geospatial + multi-source aggregation under search-like QPS. Weak answer: query PG with PostGIS radius filter. Strong answer: pre-compute a geohash index of all DCs, cache inventory in Redis with short TTL (30s). Availability query = geohash prefix lookup (O(1)) + Redis HGETALL for each DC in range
inventory in Redis is a projection, not the source of truth — it gets stale within the TTL window. This is intentional: users browsing see approximately-fresh inventory, users purchasing get a real-time DB check. The two paths have different consistency requirements and must be explicitly separated. At high QPS, even Redis can become a bottleneck — read-through cache with local in-process LRU for the hottest items (fast movers, popular items) adds another layer
The core hard problem: many users buying the last unit simultaneously
check inventory then decrement (two operations — TOCTOU race)
SELECT FOR UPDATE within a PostgreSQL transaction — atomic lock, check, and decrement
tradeoff: pessimistic locking works but creates serialization under high concurrent orders for the same item. Optimistic locking (CAS via version number) is better under low contention, worse under high contention. For a local delivery system where popular items genuinely spike (weather events, flash sales), pessimistic is the safer default. SQS queue for order submission acts as a shock absorber: workers drain the queue at a safe DB write rate, and user sees "order processing" rather than an error. Idempotency key on every order prevents double-charge on client retry
find the nearest DC by Euclidean distance
a production system doesn't just find the nearest DC by Euclidean distance — it finds the nearest DC that can fulfill the order within the delivery window. This requires: (1) drive time estimate from DC to delivery address (routing API, not just radius), (2) DC capacity check (is there a driver available?), (3) inventory check (does this DC have the items?)
this is a constraint satisfaction problem across three dimensions. The production approach is coarse + fine: geohash radius filter first (fast, eliminates 99% of DCs), then routing API call on the small candidate set (expensive, accurate). Cache routing estimates per (DC_geohash, delivery_zone) pair with 5-minute TTL to reduce routing API costs
Why the deep dives connect to the scaling problem: "Reads are broad and frequent; writes are rare but delicate." Deep dive 1 solves the read problem. Deep dive 2 solves the write correctness problem. Deep dive 3 solves the geo-routing precision problem.
Interview script
Two-path script.
Whiteboard
+------------------+
| Client |
| Web or Mobile |
+---------+--------+
|
Availability API | Order API
|
+----------------+----------------+
| |
v v
+-----------------------+ +-----------------------+
| Availability Service | | Orders Service |
| read path | | write path |
+----------+------------+ +-----------+-----------+
| |
| asks for serviceable DCs | asks for serviceable DCs
v v
+---------------------------+
| Nearby Service |
| find DCs within 1 hour |
+-------------+-------------+
|
| candidate DCs
v
+---------------------------+
| Travel Time Service |
| external ETA estimation |
+---------------------------+
Availability read flow
----------------------
|
v
+-----------------------+
| Redis Cache |
| availability results |
| short TTL |
+----------+------------+
|
cache miss
v
+-----------------------+
| Postgres Read Replica |
| inventory reads |
+----------+------------+
|
v
+-----------------------+
| Partitioned by Region |
| inventory + items |
+-----------------------+
Order write flow
----------------
|
v
+-----------------------+
| Postgres Leader |
| serializable txn |
+----------+------------+
|
v
+-----------------------+
| Tables |
| Inventory |
| Items |
| Orders |
| OrderItems |
+-----------------------+
|
v
+-----------------------+
| Cache Invalidation |
| expire affected keys |
+-----------------------+The mental model is simple. Availability is a fast read path that can tolerate slight staleness, so it uses Nearby Service, cache, and read replicas. Orders are the strict write path, so they go to the Postgres leader in one transaction so you do not double sell inventory.
If you were drawing this in an interview, I would show just these boxes first. Then I would say reads go through cache and replicas, while writes go through the leader with an atomic transaction.
Hamming dist < 3
Organizing news from thousands of publishers into one fast scrollable feed. The system collects articles, stores metadata, and redirects users to the publisher site on click.
The challenge: fast aggregation + deduplication at scale, not hosting full articles.
The hard part is that both sides scale at once. You are ingesting articles from thousands of publishers, while also serving a huge read-heavy feed to millions of users with very fresh content.
There are three main pain points. First, feed reads are massive, especially during breaking news, so you cannot query the database for every request and still stay under 200ms. Second, the feed keeps changing while users scroll, which makes simple page-number pagination cause duplicates or missed articles. Third, freshness matters a lot, so ingestion is not just batch ETL work. You need to discover new articles quickly, update regional feeds fast, and keep caches fresh without overwhelming publishers or your own systems.
- Two pipelines. Write pipeline ingests from publishers. Read pipeline serves cached feeds to users.
- Collect. RSS, publisher APIs, or web scraping.
- Dedup. SimHash fingerprint per article. Hamming distance < 3 = near-duplicate, discard.
- Precompute feeds. Regional Redis sorted sets. New articles update them async. Reads are fast cache lookups.
- Pagination. Cursor-based, not page numbers. Offset pagination causes gaps when new articles arrive.
- Thumbnails. Copy to object storage + CDN rather than hotlinking publisher images.
query Cassandra per request, assemble feed on-demand
the scaling pain is that you can't query the database for every feed request at 11,600 QPS and stay under 200ms. Weak answer: query Cassandra per request. Strong answer: pre-compute regional Redis sorted sets (score = publish timestamp) and serve feeds from cache. The hard part is keeping pre-computed feeds fresh during breaking news when articles arrive at high frequency
write-through feed population — when an article passes dedup and is stored, the ingestion service immediately pushes its ID into the relevant regional sorted sets. This inverts the cache pattern: writes are the fan-out, reads are O(1) ZREVRANGE. Tradeoffs to name: (1) feed storage cost (100M users × 50 IDs × 8 bytes = 40 GB Redis — clusters needed), (2) feed staleness during cache update, (3) thundering herd if a regional cache expires simultaneously. Fix the last one with probabilistic early expiration
Offset pagination is broken for live feeds: new articles shift existing positions while the user scrolls, causing duplicates (article appears twice) or gaps (article skipped)
use offset pagination, acknowledge the issue
cursor pagination using a composite cursor (published_at + article_id). The cursor encodes where the user is in the feed at the moment they fetched the previous page — new articles added to the top don't shift relative positions below the cursor
cursor must be stable under concurrent feed updates. Using (published_at, article_id) as cursor is stable because we sort by published_at DESC, article_id DESC — inserting new articles at the top doesn't reorder articles below any existing cursor position. Implementation: SELECT * FROM feed WHERE (published_at, article_id) < (cursor_time, cursor_id) ORDER BY published_at DESC, article_id DESC LIMIT 20
use exact SHA-256 hashing — only catches identical articles
SimHash dedup: each article gets a fingerprint by tokenizing text → compute tf-idf weights → hash weighted term vector to 64-bit integer. Hamming distance < 3 between two fingerprints = near-duplicate. This catches the same story reported by 50 publishers with slight wording differences
SimHash lookup requires comparing a new fingerprint against all stored fingerprints — at 1M articles/day this is O(N) per insert. The solution: partition fingerprints by their first K bits (locality-sensitive hashing) so only fingerprints in the same bucket need comparison. Story clustering goes further: group near-duplicates into a "story" entity with the highest-authority source as the canonical article. Ranking: recency × source authority score (PageRank-like, precomputed per domain)
Why the deep dives connect to the scaling problem: "Both sides scale at once — ingestion and reads." Deep dive 1 solves read scaling. Deep dive 2 solves pagination correctness. Deep dive 3 solves the content quality problem that makes the system valuable.
+----------------------+
| News Publishers |
| RSS APIs Webhooks |
+----------+-----------+
|
v
+-------------------------------+
| Data Collection Service |
| poll feeds parse ingest media |
+---------------+---------------+
|
+------------------+------------------+
| |
v v
+----------------------+ +----------------------+
| Article Database | | Object Storage |
| articles publishers | | thumbnails images |
+----------+-----------+ +----------+-----------+
| |
| new article writes |
v v
+----------------------+ +----------------------+
| CDC Event Stream | | CDN |
| change notifications | | serve thumbnails |
+----------+-----------+ +----------------------+
|
v
+------------------------------+
| Feed Generation Workers |
| update regional feed caches |
+--------------+---------------+
|
v
+------------------------------+
| Redis Feed Cache |
| feed:US feed:UK sorted sets |
| recent article ids by time |
+--------------+---------------+
|
v
+-----------+ +----------------------+ +------------------+
| Client |-->| API Gateway |-->| Feed Service |
| web mobile| | auth rate limit | | get feed paginate|
+-----------+ +----------------------+ +---------+--------+
|
+--------------------+-------------------+
| |
v v
+----------------------+ +----------------------+
| Redis Feed Cache | | Article Database |
| primary read path | | cache miss fallback |
+----------------------+ +----------------------+
|
v
+------------------+
| Feed Response |
| title summary |
| thumbnail url |
| publisher url |
+------------------+The main idea is two pipelines. One pipeline ingests articles from publishers and stores article metadata plus thumbnails. The other pipeline serves users by reading precomputed regional feeds from Redis so feed requests stay fast.
If you want the best interview version, draw the high level boxes first, then call out one improvement. Use cursor pagination for infinite scroll and Redis precomputed regional feeds for low latency. That shows the core system clearly without overcrowding the board.
Problem
Organizing news from thousands of publishers into one fast scrollable feed. The system collects articles, stores metadata, and redirects users to the publisher site on click.
The challenge: fast aggregation + deduplication at scale, not hosting full articles.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is that both sides scale at once. You are ingesting articles from thousands of publishers, while also serving a huge read-heavy feed to millions of users with very fresh content.
There are three main pain points. First, feed reads are massive, especially during breaking news, so you cannot query the database for every request and still stay under 200ms. Second, the feed keeps changing while users scroll, which makes simple page-number pagination cause duplicates or missed articles. Third, freshness matters a lot, so ingestion is not just batch ETL work. You need to discover new articles quickly, update regional feeds fast, and keep caches fresh without overwhelming publishers or your own systems.
Key points
- Two pipelines. Write pipeline ingests from publishers. Read pipeline serves cached feeds to users.
- Collect. RSS, publisher APIs, or web scraping.
- Dedup. SimHash fingerprint per article. Hamming distance < 3 = near-duplicate, discard.
- Precompute feeds. Regional Redis sorted sets. New articles update them async. Reads are fast cache lookups.
- Pagination. Cursor-based, not page numbers. Offset pagination causes gaps when new articles arrive.
- Thumbnails. Copy to object storage + CDN rather than hotlinking publisher images.
Tradeoffs
Deep dives
query Cassandra per request, assemble feed on-demand
the scaling pain is that you can't query the database for every feed request at 11,600 QPS and stay under 200ms. Weak answer: query Cassandra per request. Strong answer: pre-compute regional Redis sorted sets (score = publish timestamp) and serve feeds from cache. The hard part is keeping pre-computed feeds fresh during breaking news when articles arrive at high frequency
write-through feed population — when an article passes dedup and is stored, the ingestion service immediately pushes its ID into the relevant regional sorted sets. This inverts the cache pattern: writes are the fan-out, reads are O(1) ZREVRANGE. Tradeoffs to name: (1) feed storage cost (100M users × 50 IDs × 8 bytes = 40 GB Redis — clusters needed), (2) feed staleness during cache update, (3) thundering herd if a regional cache expires simultaneously. Fix the last one with probabilistic early expiration
Offset pagination is broken for live feeds: new articles shift existing positions while the user scrolls, causing duplicates (article appears twice) or gaps (article skipped)
use offset pagination, acknowledge the issue
cursor pagination using a composite cursor (published_at + article_id). The cursor encodes where the user is in the feed at the moment they fetched the previous page — new articles added to the top don't shift relative positions below the cursor
cursor must be stable under concurrent feed updates. Using (published_at, article_id) as cursor is stable because we sort by published_at DESC, article_id DESC — inserting new articles at the top doesn't reorder articles below any existing cursor position. Implementation: SELECT * FROM feed WHERE (published_at, article_id) < (cursor_time, cursor_id) ORDER BY published_at DESC, article_id DESC LIMIT 20
use exact SHA-256 hashing — only catches identical articles
SimHash dedup: each article gets a fingerprint by tokenizing text → compute tf-idf weights → hash weighted term vector to 64-bit integer. Hamming distance < 3 between two fingerprints = near-duplicate. This catches the same story reported by 50 publishers with slight wording differences
SimHash lookup requires comparing a new fingerprint against all stored fingerprints — at 1M articles/day this is O(N) per insert. The solution: partition fingerprints by their first K bits (locality-sensitive hashing) so only fingerprints in the same bucket need comparison. Story clustering goes further: group near-duplicates into a "story" entity with the highest-authority source as the canonical article. Ranking: recency × source authority score (PageRank-like, precomputed per domain)
Why the deep dives connect to the scaling problem: "Both sides scale at once — ingestion and reads." Deep dive 1 solves read scaling. Deep dive 2 solves pagination correctness. Deep dive 3 solves the content quality problem that makes the system valuable.
Interview script
Whiteboard
+----------------------+
| News Publishers |
| RSS APIs Webhooks |
+----------+-----------+
|
v
+-------------------------------+
| Data Collection Service |
| poll feeds parse ingest media |
+---------------+---------------+
|
+------------------+------------------+
| |
v v
+----------------------+ +----------------------+
| Article Database | | Object Storage |
| articles publishers | | thumbnails images |
+----------+-----------+ +----------+-----------+
| |
| new article writes |
v v
+----------------------+ +----------------------+
| CDC Event Stream | | CDN |
| change notifications | | serve thumbnails |
+----------+-----------+ +----------------------+
|
v
+------------------------------+
| Feed Generation Workers |
| update regional feed caches |
+--------------+---------------+
|
v
+------------------------------+
| Redis Feed Cache |
| feed:US feed:UK sorted sets |
| recent article ids by time |
+--------------+---------------+
|
v
+-----------+ +----------------------+ +------------------+
| Client |-->| API Gateway |-->| Feed Service |
| web mobile| | auth rate limit | | get feed paginate|
+-----------+ +----------------------+ +---------+--------+
|
+--------------------+-------------------+
| |
v v
+----------------------+ +----------------------+
| Redis Feed Cache | | Article Database |
| primary read path | | cache miss fallback |
+----------------------+ +----------------------+
|
v
+------------------+
| Feed Response |
| title summary |
| thumbnail url |
| publisher url |
+------------------+The main idea is two pipelines. One pipeline ingests articles from publishers and stores article metadata plus thumbnails. The other pipeline serves users by reading precomputed regional feeds from Redis so feed requests stay fast.
If you want the best interview version, draw the high level boxes first, then call out one improvement. Use cursor pagination for infinite scroll and Redis precomputed regional feeds for low latency. That shows the core system clearly without overcrowding the board.
Selling seats for a live event without double-booking, while millions of people browse, search, and buy simultaneously.
The hard part: high availability for browsing but strong consistency for booking — one seat can only be sold once.
The hard part is contention under huge spikes. Ticketmaster is not just a read heavy system. It is a system where millions of people may fight over the exact same tiny set of seats at the same moment.
That creates three main scaling pain points. First, the event page and seat map get hammered by refresh traffic, so reads spike hard and cached data goes stale fast. Second, booking is a hotspot write problem because many users try to reserve the same seat, so you need very careful coordination to avoid double booking. Third, search and queueing have to stay responsive during the surge, or users will overload the system before they even reach checkout. A good mental model is broad traffic everywhere, but extreme contention at a few hot seats.
- 3 core flows. View events, search events, book tickets.
- Core tension. Browsing wants high availability. Booking wants strong consistency.
- Booking pattern. Reserve seat (Redis TTL hold) → user pays → confirm (PG commit). Never hold a DB transaction open for minutes.
- Scale reads. Cache event details. Elasticsearch for search, not SQL LIKE scans.
- Peak demand. SSE for real-time seat map updates. Virtual waiting room for extreme onsales.
This is the defining hard problem for Ticketmaster. The scenario: Taylor Swift onsale, 2M users, 50K seats, all trying to book in 60 seconds
database SELECT + UPDATE
Redis SETNX per seat_id with 10-minute TTL creates a soft hold atomically (first caller wins, Redis is single-threaded so no race). On payment success, a short PG transaction commits the hard booking
the two-phase design explicitly separates hold duration (user has time to pay) from commit duration (DB transaction is < 100ms). Never hold a DB transaction open for 10 minutes — lock contention at scale would serialize all booking requests. TTL auto-release eliminates abandoned carts without any cleanup job. Key failure mode to proactively surface: Redis hold state and PG booking state can diverge if a crash occurs between them — on recovery, reconcile by querying PG for completed payments and releasing any Redis holds for unpaid seats
During an onsale, users need to see seat availability update in near-real-time as others hold and release seats
HTTP polling every 5 seconds
SSE for server-push updates, Kafka for event distribution, delivery servers partitioned by event_id
SSE fan-out math matters — 2M users watching the same event × 1 update/hold event = massive write amplification. Mitigation: (1) coalescing — batch seat status changes into 500ms update windows rather than per-event pushes; (2) broadcast the full seat map diff rather than individual seat changes; (3) for extreme events, short TTL CDN caching of the seat map image reduces real-time update pressure. The seat map itself is event-specific static data — cache aggressively. Only availability state is dynamic
Without a queue, 2M simultaneous users hit booking flow in 60 seconds = 33K booking QPS
scale horizontally
virtual waiting room caps entry into booking flow at a rate the system can handle (e.g., 5K users/minute). Ticket purchasing is a funnel: users in waiting room → hold seat → payment → confirmation. The queue smooths the spike
design: waiting room assigns users a randomized position (not FIFO — prevents queue jumping bots who connect milliseconds early). Position is stored in Redis sorted set with score = random. Users poll their position. When it's their turn, they receive a signed token that grants entry to booking flow (token has 5-minute TTL). The token prevents users from sharing their queue position. Estimate: with 2M users and a 5K/min entry rate, median wait ≈ 200 minutes. For popular events this is expected and communicated to users upfront
Why the deep dives connect to the scaling problem: "Extreme contention at a few hot seats." Deep dive 1 solves booking correctness. Deep dive 2 solves real-time UX. Deep dive 3 solves traffic smoothing.
+-------------------+
| Clients |
| Web / Mobile |
+---------+---------+
|
HTTPS|
v
+-------------------+
| Load Balancer |
+---------+---------+
|
v
+-------------------+
| API Gateway |
| auth, rate limit |
+----+----+----+----+
| | |
----------------+ | +----------------
| | |
v v v
+----------------+ +----------------+ +----------------+
| Event Service | | Search Service | | Booking Service|
+--------+-------+ +--------+-------+ +---+--------+---+
| | | |
| | | |
v v v v
+----------------+ +----------------+ +------+ +----------------+
| Cache Redis | | Elasticsearch | |Redis | | Payment |
| event details | | full text | |Locks | | Processor |
+--------+-------+ +--------+-------+ | TTL | | Stripe |
| ^ +--+---+ +--------+-------+
| | | |
v | | |
+---------------------------------------------------------------+
| PostgreSQL |
| Events | Venues | Performers | Tickets | Bookings | Users |
+---------------------------------------------------------------+
^ ^
| |
+---------+----------+
|
CDC / sync
|
v
+------------------+
| Search indexer |
| or CDC pipeline |
+------------------+
Real-time updates for seat map
+----------------+
| Realtime/SSE |
| update service |
+-------+--------+
|
v
push seat status changes to clients
Optional protection for huge onsales
+----------------------+
| Virtual waiting queue|
| Redis sorted set |
+----------+-----------+
|
v
admits limited users to booking flowThe main idea is simple. Reads go through Event Service and Search Service, and writes with contention go through Booking Service. You cache event data for heavy reads, use Elasticsearch for fast search, use Redis TTL locks for temporary seat holds, and use PostgreSQL as the source of truth so you never double book.
If you want, I can also show you a smaller interview-ready version that is easier to draw in 2 minutes.
Problem
Selling seats for a live event without double-booking, while millions of people browse, search, and buy simultaneously.
The hard part: high availability for browsing but strong consistency for booking — one seat can only be sold once.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is contention under huge spikes. Ticketmaster is not just a read heavy system. It is a system where millions of people may fight over the exact same tiny set of seats at the same moment.
That creates three main scaling pain points. First, the event page and seat map get hammered by refresh traffic, so reads spike hard and cached data goes stale fast. Second, booking is a hotspot write problem because many users try to reserve the same seat, so you need very careful coordination to avoid double booking. Third, search and queueing have to stay responsive during the surge, or users will overload the system before they even reach checkout. A good mental model is broad traffic everywhere, but extreme contention at a few hot seats.
Key points
- 3 core flows. View events, search events, book tickets.
- Core tension. Browsing wants high availability. Booking wants strong consistency.
- Booking pattern. Reserve seat (Redis TTL hold) → user pays → confirm (PG commit). Never hold a DB transaction open for minutes.
- Scale reads. Cache event details. Elasticsearch for search, not SQL LIKE scans.
- Peak demand. SSE for real-time seat map updates. Virtual waiting room for extreme onsales.
Tradeoffs
Deep dives
This is the defining hard problem for Ticketmaster. The scenario: Taylor Swift onsale, 2M users, 50K seats, all trying to book in 60 seconds
database SELECT + UPDATE
Redis SETNX per seat_id with 10-minute TTL creates a soft hold atomically (first caller wins, Redis is single-threaded so no race). On payment success, a short PG transaction commits the hard booking
the two-phase design explicitly separates hold duration (user has time to pay) from commit duration (DB transaction is < 100ms). Never hold a DB transaction open for 10 minutes — lock contention at scale would serialize all booking requests. TTL auto-release eliminates abandoned carts without any cleanup job. Key failure mode to proactively surface: Redis hold state and PG booking state can diverge if a crash occurs between them — on recovery, reconcile by querying PG for completed payments and releasing any Redis holds for unpaid seats
During an onsale, users need to see seat availability update in near-real-time as others hold and release seats
HTTP polling every 5 seconds
SSE for server-push updates, Kafka for event distribution, delivery servers partitioned by event_id
SSE fan-out math matters — 2M users watching the same event × 1 update/hold event = massive write amplification. Mitigation: (1) coalescing — batch seat status changes into 500ms update windows rather than per-event pushes; (2) broadcast the full seat map diff rather than individual seat changes; (3) for extreme events, short TTL CDN caching of the seat map image reduces real-time update pressure. The seat map itself is event-specific static data — cache aggressively. Only availability state is dynamic
Without a queue, 2M simultaneous users hit booking flow in 60 seconds = 33K booking QPS
scale horizontally
virtual waiting room caps entry into booking flow at a rate the system can handle (e.g., 5K users/minute). Ticket purchasing is a funnel: users in waiting room → hold seat → payment → confirmation. The queue smooths the spike
design: waiting room assigns users a randomized position (not FIFO — prevents queue jumping bots who connect milliseconds early). Position is stored in Redis sorted set with score = random. Users poll their position. When it's their turn, they receive a signed token that grants entry to booking flow (token has 5-minute TTL). The token prevents users from sharing their queue position. Estimate: with 2M users and a 5K/min entry rate, median wait ≈ 200 minutes. For popular events this is expected and communicated to users upfront
Why the deep dives connect to the scaling problem: "Extreme contention at a few hot seats." Deep dive 1 solves booking correctness. Deep dive 2 solves real-time UX. Deep dive 3 solves traffic smoothing.
Interview script
Whiteboard
+-------------------+
| Clients |
| Web / Mobile |
+---------+---------+
|
HTTPS|
v
+-------------------+
| Load Balancer |
+---------+---------+
|
v
+-------------------+
| API Gateway |
| auth, rate limit |
+----+----+----+----+
| | |
----------------+ | +----------------
| | |
v v v
+----------------+ +----------------+ +----------------+
| Event Service | | Search Service | | Booking Service|
+--------+-------+ +--------+-------+ +---+--------+---+
| | | |
| | | |
v v v v
+----------------+ +----------------+ +------+ +----------------+
| Cache Redis | | Elasticsearch | |Redis | | Payment |
| event details | | full text | |Locks | | Processor |
+--------+-------+ +--------+-------+ | TTL | | Stripe |
| ^ +--+---+ +--------+-------+
| | | |
v | | |
+---------------------------------------------------------------+
| PostgreSQL |
| Events | Venues | Performers | Tickets | Bookings | Users |
+---------------------------------------------------------------+
^ ^
| |
+---------+----------+
|
CDC / sync
|
v
+------------------+
| Search indexer |
| or CDC pipeline |
+------------------+
Real-time updates for seat map
+----------------+
| Realtime/SSE |
| update service |
+-------+--------+
|
v
push seat status changes to clients
Optional protection for huge onsales
+----------------------+
| Virtual waiting queue|
| Redis sorted set |
+----------+-----------+
|
v
admits limited users to booking flowThe main idea is simple. Reads go through Event Service and Search Service, and writes with contention go through Booking Service. You cache event data for heavy reads, use Elasticsearch for fast search, use Redis TTL locks for temporary seat holds, and use PostgreSQL as the source of truth so you never double book.
If you want, I can also show you a smaller interview-ready version that is easier to draw in 2 minutes.
connections
recipient_id
connections
Large-scale real-time chat. Let users send messages with very low delay, and still receive them later if offline.
Four concerns: group chats, fast delivery over persistent WebSocket, durable storage, and media sharing via blob storage.
The hard part is that WhatsApp mixes huge connection scale with strict delivery needs. You are not just storing chats. You are keeping hundreds of millions of long lived socket connections open, routing each message to the right server fast, and still making sure offline users eventually get every message.
There are three main scaling pain points. First, connection management is massive because every online user may hold one or more persistent connections, and those users are spread across many chat servers. Second, message routing gets tricky once sender and receiver are on different servers, so you need some way to bounce messages across the fleet without losing them. Third, delivery is both real time and durable. If Redis style pub sub drops a live message, the system still needs inbox storage, acks, reconnect logic, and sync to guarantee the user eventually gets it.
A fourth issue is fan out in group chats and multi device delivery. One message may need to reach many participants and several devices per participant, which multiplies writes and delivery work. So the simple interview answer is that WhatsApp is hard because it combines real time sockets, durable messaging, cross server routing, and offline sync all at once.
- Mental model. WebSocket for now, DB for later, blob storage for files.
- Send path. Write message durably first (Cassandra + Inbox), then try real-time delivery. Durability before speed.
- Offline delivery. Inbox entry persists until client acks. On reconnect, server replays pending messages.
- Cross-server routing. Redis Pub/Sub routes events to whichever server holds a user's connection. Best-effort — durability is in DB.
- Attachments. Pre-signed URL upload directly to blob storage. Server only stores the file reference.
use a centralized message broker — every server publishes and subscribes to a shared queue
the core scaling pain is connection state: 100M online users each hold a persistent WebSocket to one of ~3,000 chat servers. A message from user A on server 1 to user B on server 3 must be routed without a central coordinator. Weak answer: Redis Pub/Sub. Strong answer: Redis Pub/Sub + per-user server affinity. Server affinity map: user_id → server_id stored in Redis hash (HSET). On connect: update map. On message send: look up recipient's server, publish to that server's channel. If recipient is offline: write to Cassandra Inbox directly
to name: what happens if the server holding a user's connection goes down? Client reconnects within 5 seconds (exponential backoff). Server affinity map is updated on reconnect. In-flight messages to the old server that weren't delivered: the Inbox ensures they're eventually delivered — real-time delivery is best-effort, durability is guaranteed by Cassandra. Never lose a message even when the delivery path fails
WhatsApp's delivery promise: messages are never lost
write to DB, deliver, done
explicit ack protocol. When A sends a message: (1) server writes to Cassandra Inbox for B, (2) server attempts real-time delivery via WebSocket, (3) server returns delivery confirmation to A. B's client acks receipt: (4) client receives message, (5) sends ack to its chat server, (6) server marks Inbox entry as delivered, (7) server relays ack to A (double-checkmark). Read receipt: B opens the conversation, client sends read event, server relays to A (blue checkmark)
design: ack messages are small (just message_id + status) and can be batched. If B goes offline mid-conversation, Inbox stores pending messages. On reconnect: server queries Inbox WHERE user_id = B AND delivered = false, sends all pending, waits for acks before marking delivered. Message ordering: Cassandra clustering key on (conversation_id, created_at, message_id) gives total order within a conversation
fan out one-by-one to all group members on every message. Naive approach: 1,000 individual deliveries per message. At 10 messages/min in an active group: 10,000 deliveries/min for one group
Weak answer: fan out one-by-one to all group members on every message. Naive approach: 1,000 individual deliveries per message. At 10 messages/min in an active group: 10,000 deliveries/min for one group
design: group chat server affinity — hash(group_id) to a dedicated set of chat servers. All group members' connections are preferentially routed to these servers. Fan-out from one message: server looks up group members, identifies which are connected to itself (direct push), which are on other servers in the affinity set (local pub/sub channel), which are offline (Cassandra Inbox). This reduces the fan-out from 1,000 individual cross-server calls to a broadcast within a small server cluster. Multi-device: Inbox is per-device, not per-user. Each device has its own Inbox entry and acks independently. Message is marked fully delivered only when all active devices have acked
Why the deep dives connect to the scaling problem: "Huge connection scale, routing, durable messaging, and group fan-out." Each deep dive addresses one layer.
+----------------------+
| Mobile and Web |
| Clients |
+----------+-----------+
|
WebSocket over TLS
|
+----------v-----------+
| L4 Load |
| Balancer |
+----------+-----------+
|
+------------------+------------------+
| |
+--------v--------+ +--------v--------+
| Chat Server | | Chat Server |
| A | | B |
|-----------------| |-----------------|
| conn map | | conn map |
| ack handling | | ack handling |
| heartbeat | | heartbeat |
| inbox sync | | inbox sync |
+---+---------+---+ +---+---------+---+
| | | |
| +-----------+ +-----------+ |
| | | |
| +------v---v------+ |
| | Redis Pub Sub | |
| | user channels | |
| +------+----------+ |
| | |
| | |
+--------v--------+ +--------v--------+ +--------v--------+
| Chat Table | | Message Table | | Inbox Table |
| chat metadata | | durable msgs | | undelivered per |
| by chatId | | by messageId | | user or client |
+-----------------+ +-----------------+ +-----------------+
\\ | /
\\ | /
\\ +---------v---------+ /
+----->| ChatParticipant |<------------+
| chatId, userId |
| + GSI by userId |
+-------------------+
Attachment flow
Client --get upload target--> Chat Server
Client --upload directly-----> Blob Storage
Client --send message with attachment URL--> Chat Server
Recipient --download with signed URL-------> Blob Storage
Delivery flowProblem
Large-scale real-time chat. Let users send messages with very low delay, and still receive them later if offline.
Four concerns: group chats, fast delivery over persistent WebSocket, durable storage, and media sharing via blob storage.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is that WhatsApp mixes huge connection scale with strict delivery needs. You are not just storing chats. You are keeping hundreds of millions of long lived socket connections open, routing each message to the right server fast, and still making sure offline users eventually get every message.
There are three main scaling pain points. First, connection management is massive because every online user may hold one or more persistent connections, and those users are spread across many chat servers. Second, message routing gets tricky once sender and receiver are on different servers, so you need some way to bounce messages across the fleet without losing them. Third, delivery is both real time and durable. If Redis style pub sub drops a live message, the system still needs inbox storage, acks, reconnect logic, and sync to guarantee the user eventually gets it.
A fourth issue is fan out in group chats and multi device delivery. One message may need to reach many participants and several devices per participant, which multiplies writes and delivery work. So the simple interview answer is that WhatsApp is hard because it combines real time sockets, durable messaging, cross server routing, and offline sync all at once.
Key points
- Mental model. WebSocket for now, DB for later, blob storage for files.
- Send path. Write message durably first (Cassandra + Inbox), then try real-time delivery. Durability before speed.
- Offline delivery. Inbox entry persists until client acks. On reconnect, server replays pending messages.
- Cross-server routing. Redis Pub/Sub routes events to whichever server holds a user's connection. Best-effort — durability is in DB.
- Attachments. Pre-signed URL upload directly to blob storage. Server only stores the file reference.
Tradeoffs
Deep dives
use a centralized message broker — every server publishes and subscribes to a shared queue
the core scaling pain is connection state: 100M online users each hold a persistent WebSocket to one of ~3,000 chat servers. A message from user A on server 1 to user B on server 3 must be routed without a central coordinator. Weak answer: Redis Pub/Sub. Strong answer: Redis Pub/Sub + per-user server affinity. Server affinity map: user_id → server_id stored in Redis hash (HSET). On connect: update map. On message send: look up recipient's server, publish to that server's channel. If recipient is offline: write to Cassandra Inbox directly
to name: what happens if the server holding a user's connection goes down? Client reconnects within 5 seconds (exponential backoff). Server affinity map is updated on reconnect. In-flight messages to the old server that weren't delivered: the Inbox ensures they're eventually delivered — real-time delivery is best-effort, durability is guaranteed by Cassandra. Never lose a message even when the delivery path fails
WhatsApp's delivery promise: messages are never lost
write to DB, deliver, done
explicit ack protocol. When A sends a message: (1) server writes to Cassandra Inbox for B, (2) server attempts real-time delivery via WebSocket, (3) server returns delivery confirmation to A. B's client acks receipt: (4) client receives message, (5) sends ack to its chat server, (6) server marks Inbox entry as delivered, (7) server relays ack to A (double-checkmark). Read receipt: B opens the conversation, client sends read event, server relays to A (blue checkmark)
design: ack messages are small (just message_id + status) and can be batched. If B goes offline mid-conversation, Inbox stores pending messages. On reconnect: server queries Inbox WHERE user_id = B AND delivered = false, sends all pending, waits for acks before marking delivered. Message ordering: Cassandra clustering key on (conversation_id, created_at, message_id) gives total order within a conversation
fan out one-by-one to all group members on every message. Naive approach: 1,000 individual deliveries per message. At 10 messages/min in an active group: 10,000 deliveries/min for one group
Weak answer: fan out one-by-one to all group members on every message. Naive approach: 1,000 individual deliveries per message. At 10 messages/min in an active group: 10,000 deliveries/min for one group
design: group chat server affinity — hash(group_id) to a dedicated set of chat servers. All group members' connections are preferentially routed to these servers. Fan-out from one message: server looks up group members, identifies which are connected to itself (direct push), which are on other servers in the affinity set (local pub/sub channel), which are offline (Cassandra Inbox). This reduces the fan-out from 1,000 individual cross-server calls to a broadcast within a small server cluster. Multi-device: Inbox is per-device, not per-user. Each device has its own Inbox entry and acks independently. Message is marked fully delivered only when all active devices have acked
Why the deep dives connect to the scaling problem: "Huge connection scale, routing, durable messaging, and group fan-out." Each deep dive addresses one layer.
Interview script
Whiteboard
+----------------------+
| Mobile and Web |
| Clients |
+----------+-----------+
|
WebSocket over TLS
|
+----------v-----------+
| L4 Load |
| Balancer |
+----------+-----------+
|
+------------------+------------------+
| |
+--------v--------+ +--------v--------+
| Chat Server | | Chat Server |
| A | | B |
|-----------------| |-----------------|
| conn map | | conn map |
| ack handling | | ack handling |
| heartbeat | | heartbeat |
| inbox sync | | inbox sync |
+---+---------+---+ +---+---------+---+
| | | |
| +-----------+ +-----------+ |
| | | |
| +------v---v------+ |
| | Redis Pub Sub | |
| | user channels | |
| +------+----------+ |
| | |
| | |
+--------v--------+ +--------v--------+ +--------v--------+
| Chat Table | | Message Table | | Inbox Table |
| chat metadata | | durable msgs | | undelivered per |
| by chatId | | by messageId | | user or client |
+-----------------+ +-----------------+ +-----------------+
\\ | /
\\ | /
\\ +---------v---------+ /
+----->| ChatParticipant |<------------+
| chatId, userId |
| + GSI by userId |
+-------------------+
Attachment flow
Client --get upload target--> Chat Server
Client --upload directly-----> Blob Storage
Client --send message with attachment URL--> Chat Server
Recipient --download with signed URL-------> Blob Storage
Delivery flowShowing each user a personalized list of recent posts from people they follow, fast and at massive scale.
The hard part: fan-out — one post can need to appear in millions of followers' feeds.
The hard part is fan-out. In FB News Feed, one user action can explode into huge work either when you read the feed or when you write a new post.
There are two core scaling pain points. First, feed reads can be expensive because to build one timeline you may need posts from a very large number of followed users, then merge and sort them fast. That is fan-out on read. Second, feed writes can also be expensive because a user with millions of followers may force you to update millions of feeds when they post. That is fan-out on write.
A third issue is skew. Most posts are quiet, but a few become very hot, so one post or one celebrity account can create uneven load on your databases and caches. So the big idea you should say in an interview is that News Feed is hard because of massive fan-out, hot keys, and the trade-off between precomputing feeds for fast reads versus computing them later for cheaper writes.
- Goal. Show each user a personalized list of recent posts from people they follow.
- Default scaling move. Precompute feeds on write. Store a bounded recent feed per user in Redis sorted set.
- Write path. New post → Cassandra → Kafka → workers update follower feeds asynchronously.
- Celebrity fix. Don't fan out to huge accounts' followers. Pull their recent posts at read time and merge.
- Hot post fix. Cache in front of post storage so viral posts don't hammer the DB.
always fan-out on write — pre-compute every follower's feed on every post
the core hard problem is fan-out: one post potentially needs to update millions of feeds. Weak answer: always fan-out on write. Strong answer: articulate the full tradeoff space. Fan-out on write: fast reads (pre-computed), expensive writes (proportional to follower count). Fan-out on read: cheap writes, expensive reads (O(followed_users) per feed request). Hybrid: fan-out on write for normal users, pull on read for celebrities above a threshold
the threshold is a configuration value, not a code decision — it should be tunable based on observed fan-out worker lag. The math: if a fan-out worker processes 10K writes/second and delivery SLA is 5 minutes, the maximum safe follower count for push fan-out is 10K × 300 = 3M. Any account above that threshold uses pull. At read time: fetch user's pre-built feed from Redis, also fetch the last N posts from each celebrity they follow from Cassandra, merge and sort. The merge is O(C × log C) where C is the number of celebrity accounts — typically small, so fast
query Cassandra directly on every feed read — join followed users' posts and sort
each user has a Redis sorted set keyed by user_id with post_ids as members and publish_timestamp as score. ZREVRANGE returns the feed in reverse-chronological order in O(log N + K) time
concerns: (1) memory: 3B users × 1,000 post IDs × 8 bytes = 24 TB of Redis storage — requires a large Redis cluster with sharding by user_id. Cost is significant — only store the most recent 1,000 posts per user (ZREMRANGEBYRANK after every ZADD to trim). (2) Hot keys: users followed by millions of other users have their post IDs written to millions of sorted sets concurrently — this is the fan-out bottleneck, not the sorted set read. (3) Cold users: users who haven't logged in for 30 days don't need a live Redis feed — evict cold feed entries, rebuild from Cassandra on next login
In production, chronological feed is deprecated in favor of ML-ranked feed
Query the database on every feed request.
In production, chronological feed is deprecated in favor of ML-ranked feed
this is a two-stage architecture. Stage 1 (retrieval): pull top-N candidates from Redis sorted set — fast, based on recency signal only. Stage 2 (ranking): pass candidates to a separate ranking service that scores each post using features (engagement rate, relationship strength, content type, recency, user interest signals). Serves top-K ranked results. These two stages are explicitly separate services with separate scaling characteristics: retrieval is read-heavy and latency-critical, ranking is compute-heavy and can tolerate 50-100ms. Staff+ architectural point: never embed ML ranking logic in the feed service — they change at different cadences, are owned by different teams, and have different failure modes. The interface is clean: retrieval service returns candidates, ranking service returns scores
Why the deep dives connect to the scaling problem: "Massive fan-out and hot keys." Deep dive 1 solves fan-out architecture. Deep dive 2 solves storage and retrieval. Deep dive 3 solves the product quality layer on top.
Clients
|
v
API Gateway / Load Balancer
|
+-------------------+-------------------+-------------------+
| | |
v v v
Post Service Follow Service Feed Service
| | |
| | |
v v v
Post Table Follow Table Precomputed Feed Table
DynamoDB DynamoDB DynamoDB
PK postId PK userFollowing PK userId
SK userFollowed value recent postIds
GSI userFollowed
Post Table GSI
PK creatorId
SK createdAt
Write path for new post
-----------------------
User -> Post Service -> Post Table
-> Queue message with postId, creatorId
v
SQS / Queue
|
v
Feed Workers
|
+--------------+--------------+
| |
v v
Follow Table GSI Precomputed Feed Table
get followers prepend new postId
Read path for feed
------------------
User -> Feed Service
-> read precomputed feed for user
-> for non-precomputed celebrity accounts, query recent posts by creatorId from Post Table GSI
-> fetch post objects by postId
-> merge + sort by createdAt
-> return page with next cursor
Hot post read protection
------------------------
Feed Service
|
v
Replicated Redis Cache
|
v
Post TableIf you want the best interview version, I would say this out loud as one sentence. Most users read from a precomputed feed, most posts are fanned out asynchronously on write, and celebrity accounts fall back to partial fan-out on read.
If you want, I can also give you a smaller interview-sized sketch that fits in 10 to 12 lines.
Problem
Showing each user a personalized list of recent posts from people they follow, fast and at massive scale.
The hard part: fan-out — one post can need to appear in millions of followers' feeds.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is fan-out. In FB News Feed, one user action can explode into huge work either when you read the feed or when you write a new post.
There are two core scaling pain points. First, feed reads can be expensive because to build one timeline you may need posts from a very large number of followed users, then merge and sort them fast. That is fan-out on read. Second, feed writes can also be expensive because a user with millions of followers may force you to update millions of feeds when they post. That is fan-out on write.
A third issue is skew. Most posts are quiet, but a few become very hot, so one post or one celebrity account can create uneven load on your databases and caches. So the big idea you should say in an interview is that News Feed is hard because of massive fan-out, hot keys, and the trade-off between precomputing feeds for fast reads versus computing them later for cheaper writes.
Key points
- Goal. Show each user a personalized list of recent posts from people they follow.
- Default scaling move. Precompute feeds on write. Store a bounded recent feed per user in Redis sorted set.
- Write path. New post → Cassandra → Kafka → workers update follower feeds asynchronously.
- Celebrity fix. Don't fan out to huge accounts' followers. Pull their recent posts at read time and merge.
- Hot post fix. Cache in front of post storage so viral posts don't hammer the DB.
Tradeoffs
Deep dives
always fan-out on write — pre-compute every follower's feed on every post
the core hard problem is fan-out: one post potentially needs to update millions of feeds. Weak answer: always fan-out on write. Strong answer: articulate the full tradeoff space. Fan-out on write: fast reads (pre-computed), expensive writes (proportional to follower count). Fan-out on read: cheap writes, expensive reads (O(followed_users) per feed request). Hybrid: fan-out on write for normal users, pull on read for celebrities above a threshold
the threshold is a configuration value, not a code decision — it should be tunable based on observed fan-out worker lag. The math: if a fan-out worker processes 10K writes/second and delivery SLA is 5 minutes, the maximum safe follower count for push fan-out is 10K × 300 = 3M. Any account above that threshold uses pull. At read time: fetch user's pre-built feed from Redis, also fetch the last N posts from each celebrity they follow from Cassandra, merge and sort. The merge is O(C × log C) where C is the number of celebrity accounts — typically small, so fast
query Cassandra directly on every feed read — join followed users' posts and sort
each user has a Redis sorted set keyed by user_id with post_ids as members and publish_timestamp as score. ZREVRANGE returns the feed in reverse-chronological order in O(log N + K) time
concerns: (1) memory: 3B users × 1,000 post IDs × 8 bytes = 24 TB of Redis storage — requires a large Redis cluster with sharding by user_id. Cost is significant — only store the most recent 1,000 posts per user (ZREMRANGEBYRANK after every ZADD to trim). (2) Hot keys: users followed by millions of other users have their post IDs written to millions of sorted sets concurrently — this is the fan-out bottleneck, not the sorted set read. (3) Cold users: users who haven't logged in for 30 days don't need a live Redis feed — evict cold feed entries, rebuild from Cassandra on next login
In production, chronological feed is deprecated in favor of ML-ranked feed
Query the database on every feed request.
In production, chronological feed is deprecated in favor of ML-ranked feed
this is a two-stage architecture. Stage 1 (retrieval): pull top-N candidates from Redis sorted set — fast, based on recency signal only. Stage 2 (ranking): pass candidates to a separate ranking service that scores each post using features (engagement rate, relationship strength, content type, recency, user interest signals). Serves top-K ranked results. These two stages are explicitly separate services with separate scaling characteristics: retrieval is read-heavy and latency-critical, ranking is compute-heavy and can tolerate 50-100ms. Staff+ architectural point: never embed ML ranking logic in the feed service — they change at different cadences, are owned by different teams, and have different failure modes. The interface is clean: retrieval service returns candidates, ranking service returns scores
Why the deep dives connect to the scaling problem: "Massive fan-out and hot keys." Deep dive 1 solves fan-out architecture. Deep dive 2 solves storage and retrieval. Deep dive 3 solves the product quality layer on top.
Interview script
Whiteboard
Clients
|
v
API Gateway / Load Balancer
|
+-------------------+-------------------+-------------------+
| | |
v v v
Post Service Follow Service Feed Service
| | |
| | |
v v v
Post Table Follow Table Precomputed Feed Table
DynamoDB DynamoDB DynamoDB
PK postId PK userFollowing PK userId
SK userFollowed value recent postIds
GSI userFollowed
Post Table GSI
PK creatorId
SK createdAt
Write path for new post
-----------------------
User -> Post Service -> Post Table
-> Queue message with postId, creatorId
v
SQS / Queue
|
v
Feed Workers
|
+--------------+--------------+
| |
v v
Follow Table GSI Precomputed Feed Table
get followers prepend new postId
Read path for feed
------------------
User -> Feed Service
-> read precomputed feed for user
-> for non-precomputed celebrity accounts, query recent posts by creatorId from Post Table GSI
-> fetch post objects by postId
-> merge + sort by createdAt
-> return page with next cursor
Hot post read protection
------------------------
Feed Service
|
v
Replicated Redis Cache
|
v
Post TableIf you want the best interview version, I would say this out loud as one sentence. Most users read from a precomputed feed, most posts are fanned out asynchronously on write, and celebrity accounts fall back to partial fan-out on read.
If you want, I can also give you a smaller interview-sized sketch that fits in 10 to 12 lines.
mutual like
+ pref filter
to both users
Tinder solves real-time recommendation and mutual match detection. Show relevant nearby profiles fast, record huge swipe volumes, and reliably detect when two users both say yes.
The hard part is that Tinder has both a fast read problem and a correctness problem. You need to generate a fresh stack of nearby profiles in well under a second, while also making sure swipes and matches are recorded correctly.
There are three main scaling pain points. First, feed generation is expensive because it mixes filters like age and preferences with geospatial search, which means a simple database query gets slow fast at large scale. Second, swiping is a huge write stream, and match creation is tricky because two users can swipe on each other at nearly the same time, so you need low latency plus strong enough consistency to not miss a match. Third, you must avoid re-showing profiles a user already swiped on, which gets harder as each user builds a large swipe history.
A good mental model is this. Tinder is hard because it combines search, real-time decisions, and deduping in one loop. You are not just storing profiles. You are constantly finding nearby candidates, filtering out old ones, and atomically detecting mutual likes.
- Scope it first. Core: swipe, match, chat stub, nearby profiles. Out of scope unless asked: payments, Super Likes, video, reporting. Geo is central — say so upfront.
- Build the deck fast. Pre-computed candidate stack per user. Async re-fill from GEORADIUS + preference filter when stack runs low. Never compute deck on every swipe.
- High write volume — Cassandra. 58K swipe writes/sec. Cassandra append-only, partitioned by user_id — the right DB for this access pattern. Never use PostgreSQL for raw swipe writes at this scale.
- Instant match detection — Redis SETNX. SETNX on key min(a,b):max(a,b). Atomic. First caller creates the key. Second caller finds it exists = match. No race possible.
- Swipe dedup — Bloom filter. 73K swipes/user over 2 years. Bloom filter: O(1) check, ~730 KB per user. 1% false positive acceptable for deck (skip an unseen profile). Exact Cassandra check only for match creation.
- Privacy by design. Never expose who liked you without a mutual match. The SETNX key reveals nothing — it only exists on mutual. The pending swipe in the other direction is never returned by any API.
- Failure mode to name. Redis goes down → fall back to Cassandra LWT (Lightweight Transaction) for match detection. Slower but correct. Always have a fallback for the atomic match check.
run GEORADIUS + preference filter on every swipe in real-time
the scaling pain is that feed generation mixes geospatial search, preference filtering, and swipe history deduplication into one pipeline that must complete in under a second. Weak answer: GEORADIUS + filter in one query. Strong answer: pre-compute candidate decks asynchronously — GEORADIUS query + preference filter runs in background, results cached per user. When user opens the app, deck is ready instantly
the deck has a TTL (location changes, preferences update). Rather than a fixed TTL, invalidate the deck reactively: location update event → deck invalidation → async re-generation. Stack depth monitoring: when user swipes through to the last 10 profiles in their deck, trigger an async deck refresh before they run out. Candidate scoring: after GEORADIUS + preference filter, score remaining candidates using ML model (predicted swipe probability based on historical data) and serve the highest-scoring profiles first
58,000 swipe writes/second to Cassandra with one correctness requirement: two simultaneous mutual likes must be detected exactly once
check Cassandra for mutual like
Redis SETNX on a derived key (min(a,b):max(a,b)) for atomic mutual check. If A likes B and B already liked A, the key exists → match. If A likes B and B hasn't liked yet, SETNX succeeds → key set for later. When B likes A, SETNX finds the key → match detected
durability: Redis is not the source of truth. Cassandra stores all swipes durably. Redis is only for the real-time match detection signal. If Redis is down: fall back to Cassandra CAS (Compare-And-Swap using Lightweight Transactions) — slower but correct. Partition key for swipes: (user_id, created_at_bucket) — queries like "all swipes by user X" are efficient
store all swipe history in Cassandra, check each candidate with a point query. Each user has accumulated swipe history that must be filtered from their deck. At 100 swipes/day × 2 years = 73,000 swipes per active user. Checking all 73K against candidate profiles is expensive
Weak answer: store all swipe history in Cassandra, check each candidate with a point query. Each user has accumulated swipe history that must be filtered from their deck. At 100 swipes/day × 2 years = 73,000 swipes per active user. Checking all 73K against candidate profiles is expensive
solution: Bloom filter per user (probabilistic, O(1) check, ~10 bytes/entry at 1% false positive rate). 73K swipes × 10 bytes = 730 KB per user — stored in Redis. False positive means occasionally hiding an unseen profile (user never sees it) — this is acceptable, better than showing an already-swiped profile. Critical distinction: Bloom filter is for deck generation only. For match detection, exact Cassandra check is required — a false negative (missing a match) is unacceptable. Cuckoo filter as an upgrade: supports deletions (allowing swipe undo) at slightly higher memory cost
Why the deep dives connect to the scaling problem: "Fast read, correctness, and deduplication in one loop." Deep dives address each dimension of that loop.
Three-problem script.
+-------------------+
| Mobile Client |
| profile, feed, |
| swipe, match UI |
+---------+---------+
|
HTTPS |
v
+-------------------+
| API Gateway |
| auth, routing |
+----+---------+----+
| |
+------------+ +-----------------+
| |
v v
+---------------------+ +----------------------+
| Profile Service | | Swipe Service |
| prefs, profile data | | record swipe, detect |
+----------+----------+ | match, emit events |
| +----+-----------+-----+
| | |
v | |
+---------------------+ | v
| User DB | | +------------------+
| profiles, prefs | | | Notification Svc |
+---------------------+ | | APNS or FCM |
| +--------+---------+
| |
| v
| +---------------+
| | Other User |
| | push device |
| +---------------+
|
v
+----------------------+
| Redis Match Store |
| atomic pair check |
| low latency match |
+----------+-----------+
|
v
+----------------------+
| Swipe DB |
| Cassandra style |
| durable swipe log |
+----------------------+
Feed generation path
+---------------------+
| Feed Service |
| build candidate set |
+-----+---------+-----+
| |
| v
| +----------------------+
| | Feed Cache |
| | precomputed stacks |
| +----------------------+
|
v
+-------------------------+
| Search Index |
| geo plus preference |
| filtering |
+-----------+-------------+
|
v
+---------------------+
| User DB / CDC sync |
| profile updates flow |
| into search index |
+---------------------+The mental model is two main paths. One path serves profiles fast through feed cache plus a search index. The other path handles swipes safely through Redis for atomic match detection and Cassandra for durable swipe history.
If you want, I can also give you a simpler interview version with only 6 boxes, or a step by step swipe flow sketch.
Problem
Tinder solves real-time recommendation and mutual match detection. Show relevant nearby profiles fast, record huge swipe volumes, and reliably detect when two users both say yes.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is that Tinder has both a fast read problem and a correctness problem. You need to generate a fresh stack of nearby profiles in well under a second, while also making sure swipes and matches are recorded correctly.
There are three main scaling pain points. First, feed generation is expensive because it mixes filters like age and preferences with geospatial search, which means a simple database query gets slow fast at large scale. Second, swiping is a huge write stream, and match creation is tricky because two users can swipe on each other at nearly the same time, so you need low latency plus strong enough consistency to not miss a match. Third, you must avoid re-showing profiles a user already swiped on, which gets harder as each user builds a large swipe history.
A good mental model is this. Tinder is hard because it combines search, real-time decisions, and deduping in one loop. You are not just storing profiles. You are constantly finding nearby candidates, filtering out old ones, and atomically detecting mutual likes.
Key points
- Scope it first. Core: swipe, match, chat stub, nearby profiles. Out of scope unless asked: payments, Super Likes, video, reporting. Geo is central — say so upfront.
- Build the deck fast. Pre-computed candidate stack per user. Async re-fill from GEORADIUS + preference filter when stack runs low. Never compute deck on every swipe.
- High write volume — Cassandra. 58K swipe writes/sec. Cassandra append-only, partitioned by user_id — the right DB for this access pattern. Never use PostgreSQL for raw swipe writes at this scale.
- Instant match detection — Redis SETNX. SETNX on key min(a,b):max(a,b). Atomic. First caller creates the key. Second caller finds it exists = match. No race possible.
- Swipe dedup — Bloom filter. 73K swipes/user over 2 years. Bloom filter: O(1) check, ~730 KB per user. 1% false positive acceptable for deck (skip an unseen profile). Exact Cassandra check only for match creation.
- Privacy by design. Never expose who liked you without a mutual match. The SETNX key reveals nothing — it only exists on mutual. The pending swipe in the other direction is never returned by any API.
- Failure mode to name. Redis goes down → fall back to Cassandra LWT (Lightweight Transaction) for match detection. Slower but correct. Always have a fallback for the atomic match check.
Tradeoffs
Deep dives
run GEORADIUS + preference filter on every swipe in real-time
the scaling pain is that feed generation mixes geospatial search, preference filtering, and swipe history deduplication into one pipeline that must complete in under a second. Weak answer: GEORADIUS + filter in one query. Strong answer: pre-compute candidate decks asynchronously — GEORADIUS query + preference filter runs in background, results cached per user. When user opens the app, deck is ready instantly
the deck has a TTL (location changes, preferences update). Rather than a fixed TTL, invalidate the deck reactively: location update event → deck invalidation → async re-generation. Stack depth monitoring: when user swipes through to the last 10 profiles in their deck, trigger an async deck refresh before they run out. Candidate scoring: after GEORADIUS + preference filter, score remaining candidates using ML model (predicted swipe probability based on historical data) and serve the highest-scoring profiles first
58,000 swipe writes/second to Cassandra with one correctness requirement: two simultaneous mutual likes must be detected exactly once
check Cassandra for mutual like
Redis SETNX on a derived key (min(a,b):max(a,b)) for atomic mutual check. If A likes B and B already liked A, the key exists → match. If A likes B and B hasn't liked yet, SETNX succeeds → key set for later. When B likes A, SETNX finds the key → match detected
durability: Redis is not the source of truth. Cassandra stores all swipes durably. Redis is only for the real-time match detection signal. If Redis is down: fall back to Cassandra CAS (Compare-And-Swap using Lightweight Transactions) — slower but correct. Partition key for swipes: (user_id, created_at_bucket) — queries like "all swipes by user X" are efficient
store all swipe history in Cassandra, check each candidate with a point query. Each user has accumulated swipe history that must be filtered from their deck. At 100 swipes/day × 2 years = 73,000 swipes per active user. Checking all 73K against candidate profiles is expensive
Weak answer: store all swipe history in Cassandra, check each candidate with a point query. Each user has accumulated swipe history that must be filtered from their deck. At 100 swipes/day × 2 years = 73,000 swipes per active user. Checking all 73K against candidate profiles is expensive
solution: Bloom filter per user (probabilistic, O(1) check, ~10 bytes/entry at 1% false positive rate). 73K swipes × 10 bytes = 730 KB per user — stored in Redis. False positive means occasionally hiding an unseen profile (user never sees it) — this is acceptable, better than showing an already-swiped profile. Critical distinction: Bloom filter is for deck generation only. For match detection, exact Cassandra check is required — a false negative (missing a match) is unacceptable. Cuckoo filter as an upgrade: supports deletions (allowing swipe undo) at slightly higher memory cost
Why the deep dives connect to the scaling problem: "Fast read, correctness, and deduplication in one loop." Deep dives address each dimension of that loop.
Interview script
Three-problem script.
Whiteboard
+-------------------+
| Mobile Client |
| profile, feed, |
| swipe, match UI |
+---------+---------+
|
HTTPS |
v
+-------------------+
| API Gateway |
| auth, routing |
+----+---------+----+
| |
+------------+ +-----------------+
| |
v v
+---------------------+ +----------------------+
| Profile Service | | Swipe Service |
| prefs, profile data | | record swipe, detect |
+----------+----------+ | match, emit events |
| +----+-----------+-----+
| | |
v | |
+---------------------+ | v
| User DB | | +------------------+
| profiles, prefs | | | Notification Svc |
+---------------------+ | | APNS or FCM |
| +--------+---------+
| |
| v
| +---------------+
| | Other User |
| | push device |
| +---------------+
|
v
+----------------------+
| Redis Match Store |
| atomic pair check |
| low latency match |
+----------+-----------+
|
v
+----------------------+
| Swipe DB |
| Cassandra style |
| durable swipe log |
+----------------------+
Feed generation path
+---------------------+
| Feed Service |
| build candidate set |
+-----+---------+-----+
| |
| v
| +----------------------+
| | Feed Cache |
| | precomputed stacks |
| +----------------------+
|
v
+-------------------------+
| Search Index |
| geo plus preference |
| filtering |
+-----------+-------------+
|
v
+---------------------+
| User DB / CDC sync |
| profile updates flow |
| into search index |
+---------------------+The mental model is two main paths. One path serves profiles fast through feed cache plus a search index. The other path handles swipes safely through Redis for atomic match detection and Cassandra for durable swipe history.
If you want, I can also give you a simpler interview version with only 6 boxes, or a step by step swipe flow sketch.
CPU+mem limits
Safe, fast code evaluation at scale. Users browse problems, submit code, and get feedback quickly while the platform safely runs untrusted code.
The real challenge: executing user code in isolation, not storing problems.
The hard part is not storing problems. It is safely running huge bursts of user code while still returning results fast.
There are three main scaling pain points. First, code execution is CPU heavy and spiky, especially during contests, so a wave of submissions can overwhelm workers much faster than normal page reads would. Second, every submission is untrusted code, so you need strong isolation, timeouts, and resource limits, which makes execution slower and more expensive than a normal backend request. Third, live leaderboards can create a read storm if thousands of users poll every few seconds, so you do not want to rebuild rankings from the main database on every request.
A good mental model is this. LeetCode is hard because it mixes a fairly simple content app with a mini compute platform. The content side is easy to scale. The code runner and contest traffic are the parts that make it tricky.
- Scope it first. Core: submit code, execute safely, return verdict (Accepted/TLE/WA/RE), leaderboard for contests. Out of scope: code completion, collaboration, plagiarism detection.
- Submissions are async — always. Never execute user code in the request thread. POST /submit returns a submission_id immediately. Client polls GET /submissions/:id. Queue + workers is the only correct pattern.
- Sandbox is the hard part. Docker + seccomp profile: block socket syscalls (no network), hard CPU + memory cgroups, OOM kill at container level, read-only filesystem, PID limit. Name all four constraints.
- Container pre-warming. Cold-start Docker container: ~100ms. Pre-warm a pool sized to P99 submission rate. For contests: pre-scale 30 min before start. Autoscale by queue depth, not CPU.
- Per-language worker pools. Python submissions take 5-10× longer than C++. Single mixed queue starves fast languages. Separate pools per language, sized by submission volume mix.
- Leaderboard — Redis sorted set. ZADD contest:{id} score user_id. ZREVRANK for user rank in O(log N). Redis handles 10K leaderboard QPS trivially. SSE push for live updates — never poll.
- Failure mode to name. Worker crashes mid-execution: job requeues via SQS visibility timeout. Duplicate execution is safe because verdict is deterministic — same code + same tests always gives same result.
The defining problem is safely running untrusted code
Docker containers
Docker with specific security configuration: (1) seccomp profile (restrict allowed syscalls to the minimum needed — block socket, fork beyond a count, exec of new binaries), (2) no network namespace (--network none), (3) read-only filesystem, (4) cgroups for CPU time limit and memory limit, (5) OOM kill at container level
VMs give stronger isolation (separate kernel) at the cost of 5-10s startup vs. Docker's <100ms. Firecracker microVMs (used by AWS Lambda) give VM-level isolation at near-container startup speed — the best security/performance tradeoff. For a real coding judge: Docker + seccomp is the standard production answer. Name what your seccomp profile blocks explicitly: socket syscalls (no network), fork/clone beyond the PID limit (no fork bombs), mount syscalls (no filesystem escapes)
Contest submissions: 100K users × 5 submissions in 2 hours = 250K total. In the first 10 minutes of a contest, most submissions happen → spike to 5,000 QPS
scale workers
SQS queue decouples submission acceptance from execution. API accepts submission immediately (synchronous, <10ms), enqueues job, returns submission ID. Client polls GET /submissions/:id. Queue depth × avg execution time / worker count = queue latency. At 5,000 QPS with 10s avg execution and 200 workers: 5,000 × 10 / 200 = 250 second latency at peak. Fix: pre-scale workers before contest start (predictable traffic pattern). Worker autoscaling metric: SQS queue depth, not CPU
separate worker pools by language (Python workers, C++ workers) because Python submissions take 5-10× longer than C++ — mixed queue starves C++ users. Per-language queue with proportional worker allocation
During a contest, 100K users polling leaderboard every 10 seconds = 10K QPS on one sorted set (the leaderboard is a single ZREVRANGE key)
cache the leaderboard
tiered caching — Redis sorted set is the live source, but serve reads from a snapshot cached with 5-second TTL in each app server's local process cache. Push leaderboard updates via SSE to subscribed users rather than polling (eliminates 90% of reads)
the leaderboard sorted set is a hot key — all reads go to one Redis shard. For a contest with 100K participants this is manageable; for a global contest with 1M: shard the leaderboard by rank range (top 100 is served from one shard, ranks 100-1000 from another) and merge at the API layer. Alternatively: serve approximate leaderboards (top 10% exact, rest approximate from a lower-frequency snapshot) — users care most about top positions
Why the deep dives connect to the scaling problem: "Safe execution farm plus contest traffic spikes." Each deep dive addresses one dimension.
+-------------------+
| Web / Mobile |
| Client |
+---------+---------+
|
GET problems, submit code, poll
|
+---------v---------+
| API Server |
| auth from JWT |
| problem APIs |
| submit API |
| leaderboard API |
+----+---------+-----+
| |
read problems| |read submission status
| |
+---------v--+ +--v----------------+
| Problems DB | | Submissions DB |
| DynamoDB | | results, code, |
| problems, | | passed, metadata |
| test cases, | +---------+---------+
| code stubs | |
+------------ + |
|
enqueue job |
|
+---------v---------+
| Job Queue |
| SQS or similar |
+---------+---------+
|
pull job
|
+---------v---------+
| Submission Worker |
| picks runtime |
| loads problem |
| runs test harness |
+----+----------+----+
| |
execute code | | update leaderboard
| |
+---------------------v--+ +--v------------------+
| Sandboxed Containers | | Redis Sorted Set |
| python, java, js, etc | | competition ranks |
| CPU and memory limits | | fast top N reads |
| no network | +---------+-----------+
| timeout enforced | |
+-----------+------------+ |
| |
stdout or result |
| |
+-------------+-------------+
|
+--------v--------+
| API Server |
| returns status |
| to polling |
+--------+--------+
|
+--------v--------+
| Client |
| shows result |
| polls leaderboard|
+-----------------+If you want the interview version, I would draw the simpler version first. Start with Client, API Server, Problems DB, Submissions DB, Queue, Worker, Sandboxed Containers, and Redis for leaderboard. Then explain that problem reads are synchronous, but submission execution is asynchronous because code runs for seconds and needs isolation.
The main idea is simple. Reads go straight through the API. Code submission goes through a queue to workers, workers execute inside locked down containers, results are stored in the submissions database, and leaderboard reads come from Redis instead of recomputing from the database every time.
Problem
Safe, fast code evaluation at scale. Users browse problems, submit code, and get feedback quickly while the platform safely runs untrusted code.
The real challenge: executing user code in isolation, not storing problems.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is not storing problems. It is safely running huge bursts of user code while still returning results fast.
There are three main scaling pain points. First, code execution is CPU heavy and spiky, especially during contests, so a wave of submissions can overwhelm workers much faster than normal page reads would. Second, every submission is untrusted code, so you need strong isolation, timeouts, and resource limits, which makes execution slower and more expensive than a normal backend request. Third, live leaderboards can create a read storm if thousands of users poll every few seconds, so you do not want to rebuild rankings from the main database on every request.
A good mental model is this. LeetCode is hard because it mixes a fairly simple content app with a mini compute platform. The content side is easy to scale. The code runner and contest traffic are the parts that make it tricky.
Key points
- Scope it first. Core: submit code, execute safely, return verdict (Accepted/TLE/WA/RE), leaderboard for contests. Out of scope: code completion, collaboration, plagiarism detection.
- Submissions are async — always. Never execute user code in the request thread. POST /submit returns a submission_id immediately. Client polls GET /submissions/:id. Queue + workers is the only correct pattern.
- Sandbox is the hard part. Docker + seccomp profile: block socket syscalls (no network), hard CPU + memory cgroups, OOM kill at container level, read-only filesystem, PID limit. Name all four constraints.
- Container pre-warming. Cold-start Docker container: ~100ms. Pre-warm a pool sized to P99 submission rate. For contests: pre-scale 30 min before start. Autoscale by queue depth, not CPU.
- Per-language worker pools. Python submissions take 5-10× longer than C++. Single mixed queue starves fast languages. Separate pools per language, sized by submission volume mix.
- Leaderboard — Redis sorted set. ZADD contest:{id} score user_id. ZREVRANK for user rank in O(log N). Redis handles 10K leaderboard QPS trivially. SSE push for live updates — never poll.
- Failure mode to name. Worker crashes mid-execution: job requeues via SQS visibility timeout. Duplicate execution is safe because verdict is deterministic — same code + same tests always gives same result.
Tradeoffs
Deep dives
The defining problem is safely running untrusted code
Docker containers
Docker with specific security configuration: (1) seccomp profile (restrict allowed syscalls to the minimum needed — block socket, fork beyond a count, exec of new binaries), (2) no network namespace (--network none), (3) read-only filesystem, (4) cgroups for CPU time limit and memory limit, (5) OOM kill at container level
VMs give stronger isolation (separate kernel) at the cost of 5-10s startup vs. Docker's <100ms. Firecracker microVMs (used by AWS Lambda) give VM-level isolation at near-container startup speed — the best security/performance tradeoff. For a real coding judge: Docker + seccomp is the standard production answer. Name what your seccomp profile blocks explicitly: socket syscalls (no network), fork/clone beyond the PID limit (no fork bombs), mount syscalls (no filesystem escapes)
Contest submissions: 100K users × 5 submissions in 2 hours = 250K total. In the first 10 minutes of a contest, most submissions happen → spike to 5,000 QPS
scale workers
SQS queue decouples submission acceptance from execution. API accepts submission immediately (synchronous, <10ms), enqueues job, returns submission ID. Client polls GET /submissions/:id. Queue depth × avg execution time / worker count = queue latency. At 5,000 QPS with 10s avg execution and 200 workers: 5,000 × 10 / 200 = 250 second latency at peak. Fix: pre-scale workers before contest start (predictable traffic pattern). Worker autoscaling metric: SQS queue depth, not CPU
separate worker pools by language (Python workers, C++ workers) because Python submissions take 5-10× longer than C++ — mixed queue starves C++ users. Per-language queue with proportional worker allocation
During a contest, 100K users polling leaderboard every 10 seconds = 10K QPS on one sorted set (the leaderboard is a single ZREVRANGE key)
cache the leaderboard
tiered caching — Redis sorted set is the live source, but serve reads from a snapshot cached with 5-second TTL in each app server's local process cache. Push leaderboard updates via SSE to subscribed users rather than polling (eliminates 90% of reads)
the leaderboard sorted set is a hot key — all reads go to one Redis shard. For a contest with 100K participants this is manageable; for a global contest with 1M: shard the leaderboard by rank range (top 100 is served from one shard, ranks 100-1000 from another) and merge at the API layer. Alternatively: serve approximate leaderboards (top 10% exact, rest approximate from a lower-frequency snapshot) — users care most about top positions
Why the deep dives connect to the scaling problem: "Safe execution farm plus contest traffic spikes." Each deep dive addresses one dimension.
Interview script
Whiteboard
+-------------------+
| Web / Mobile |
| Client |
+---------+---------+
|
GET problems, submit code, poll
|
+---------v---------+
| API Server |
| auth from JWT |
| problem APIs |
| submit API |
| leaderboard API |
+----+---------+-----+
| |
read problems| |read submission status
| |
+---------v--+ +--v----------------+
| Problems DB | | Submissions DB |
| DynamoDB | | results, code, |
| problems, | | passed, metadata |
| test cases, | +---------+---------+
| code stubs | |
+------------ + |
|
enqueue job |
|
+---------v---------+
| Job Queue |
| SQS or similar |
+---------+---------+
|
pull job
|
+---------v---------+
| Submission Worker |
| picks runtime |
| loads problem |
| runs test harness |
+----+----------+----+
| |
execute code | | update leaderboard
| |
+---------------------v--+ +--v------------------+
| Sandboxed Containers | | Redis Sorted Set |
| python, java, js, etc | | competition ranks |
| CPU and memory limits | | fast top N reads |
| no network | +---------+-----------+
| timeout enforced | |
+-----------+------------+ |
| |
stdout or result |
| |
+-------------+-------------+
|
+--------v--------+
| API Server |
| returns status |
| to polling |
+--------+--------+
|
+--------v--------+
| Client |
| shows result |
| polls leaderboard|
+-----------------+If you want the interview version, I would draw the simpler version first. Start with Client, API Server, Problems DB, Submissions DB, Queue, Worker, Sandboxed Containers, and Redis for leaderboard. Then explain that problem reads are synchronous, but submission execution is asynchronous because code runs for seconds and needs isolation.
The main idea is simple. Reads go straight through the API. Code submission goes through a queue to workers, workers execute inside locked down containers, results are stored in the submissions database, and leaderboard reads come from Redis instead of recomputing from the database every time.
key=userId:window
userId:window_id. Token Bucket: tokens refill at rate R, each request costs 1.A rate limiter stops one user, bot, or client from sending too many requests in a short time.
The hard parts: distributed enforcement across many app servers, atomic check+decrement without race conditions, and choosing the right algorithm.
The hard part is that a rate limiter turns every incoming request into a fast, shared counter update. At small scale that sounds simple. At large scale, millions of requests all need low latency decisions, and those decisions must be consistent enough that one user cannot bypass limits just by hitting different servers.
There are three pain points you should call out. First, the state is write heavy. Every request updates a token bucket or counter, so your shared store can become the bottleneck. Second, correctness gets tricky under concurrency. Two servers can read the same remaining quota at once and both allow the request unless the whole read modify write step is atomic. Third, distribution makes it harder. If you shard by user or IP, you need all requests for that client to land on the same shard, and hot users or abusive IPs can overload one shard.
A good interview summary is this. Rate limiting is hard to scale because it needs very fast per request decisions, shared mutable state across many servers, and enough coordination to avoid race conditions without adding too much latency.
- Enforce at gateway. Single enforcement point at the API gateway layer.
- Redis for shared state. All app servers need to see the same counter.
- Lua script = atomic. Check + decrement in a single Lua script. No race condition.
- Token bucket default. Tokens refill at rate R, each request costs 1. Best default for most APIs.
- Fail behavior. If Redis is down: fail open (allow all) or fail closed (deny all). Name this explicitly.
use fixed window counting — INCR a key, expire at the window boundary
the algorithm choice has real operational tradeoffs, not just theoretical ones. Fixed window: simplest (INCR key, expire at window boundary), but the "double spend" attack — a burst of N requests at window T-1 and N requests at window T+1 = 2N requests in a 2-second window while the limit is N per window. Sliding window counter: curr_window_count + (prev_window_count × fraction_of_prev_window_elapsed) ≈ accurate sliding window at low memory cost. Token bucket: refill rate R tokens per second, cost 1 per request, bucket size B allows bursts up to B. Leaky bucket: queue requests and process at fixed rate — smoothest, but adds latency
sliding window counter for most API rate limiting (accurate, cheap). Token bucket for APIs that explicitly want to allow short bursts (e.g., batch endpoints). Fixed window only for very simple use cases where boundary bursts are acceptable. The algorithm choice should be justified against your NFRs — don't just name it
The race condition: two servers both read the same counter (value: 999), both check 999 < 1000 (pass), both increment to 1000. User gets 2× their quota. The naive GET → check → INCR is broken
use Redis transactions (MULTI/EXEC)
Redis Lua scripts — the script executes atomically on the Redis server; no other command can run between any two lines of the Lua script. The script: (1) LOLWUT or EXISTS to check the key, (2) GET the current count, (3) if count < limit, INCR and return allowed, else return denied. All as one atomic operation
why not MULTI/EXEC? MULTI/EXEC is optimistic — if another client modifies the key between WATCH and EXEC, the transaction fails and must retry. Under high concurrency this creates a hot retry loop. Lua is unconditional — no retry needed. INCR alone is atomic too, but only for the increment — the conditional check requires Lua
fail closed — return 429 to all requests when Redis is down
what happens when Redis is unavailable? This is the question that separates senior from staff. Options: (1) fail open — allow all requests. Better for user-facing APIs where availability > protection. (2) Fail closed — deny all requests with 429. Better for auth endpoints where one compromised request is worse than a temporary outage. (3) Local in-process fallback — each app server maintains its own token bucket. Allows N × (num_servers) total requests instead of N, but prevents total failure
design: the failure mode should be a configuration option, not a code decision. Different endpoints have different failure mode requirements. The rate limiter must be explicit about which mode it's in and expose this via metrics. Additionally: Redis Sentinel for HA (automatic failover in <30s); circuit breaker pattern at the rate limiter client so a slow Redis doesn't add latency to every API request
Why the deep dives connect to the scaling problem: "Fast per-request shared counter decisions." Deep dive 1 solves algorithm correctness. Deep dive 2 solves distributed atomicity. Deep dive 3 solves failure handling.
+----------------------+
| Config Service |
| rules and limits |
+----------+-----------+
|
periodic sync
|
+---------+ HTTPS +------------v-------------+
| Clients | --------------> | API Gateway / LB Layer |
| users | | auth parse + rate limit |
| IPs | | check before app traffic |
+---------+ +------+---------+---------+
| |
allow request | | reject request
| |
| v
| +----------------------+
| | 429 Response Builder |
| | limit remaining reset|
| +----------------------+
|
v
+---------+----------+
| Backend Services |
| social media APIs |
+--------------------+
Inside the gateway rate limiter path
extract client key
userId or IP or apiKey + endpoint rule
|
v
+----------+-----------+
| Shard Router |
| hash client key |
+----------+-----------+
|
v
+----------+-----------------------------------+
| Redis Cluster |
| shared bucket state across gateway instances |
| |
| shard 1 shard 2 shard 3 ... |
| +--------+ +--------+ +--------+ |
| |alice | |bob | |carol | |
| |tokens | |tokens | |tokens | |
| |refill | |refill | |refill | |
| +--------+ +--------+ +--------+ |
+-------------------+--------------------------+
|
atomic Lua script
|
v
read bucket -> refill tokens -> consume 1 -> return decision
Per shard HA
+------------------+
| Redis Primary |
+--------+---------+
|
replicate
|
+--------v---------+
| Redis Replica |
+------------------+
Request flowProblem
A rate limiter stops one user, bot, or client from sending too many requests in a short time.
The hard parts: distributed enforcement across many app servers, atomic check+decrement without race conditions, and choosing the right algorithm.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is that a rate limiter turns every incoming request into a fast, shared counter update. At small scale that sounds simple. At large scale, millions of requests all need low latency decisions, and those decisions must be consistent enough that one user cannot bypass limits just by hitting different servers.
There are three pain points you should call out. First, the state is write heavy. Every request updates a token bucket or counter, so your shared store can become the bottleneck. Second, correctness gets tricky under concurrency. Two servers can read the same remaining quota at once and both allow the request unless the whole read modify write step is atomic. Third, distribution makes it harder. If you shard by user or IP, you need all requests for that client to land on the same shard, and hot users or abusive IPs can overload one shard.
A good interview summary is this. Rate limiting is hard to scale because it needs very fast per request decisions, shared mutable state across many servers, and enough coordination to avoid race conditions without adding too much latency.
Key points
- Enforce at gateway. Single enforcement point at the API gateway layer.
- Redis for shared state. All app servers need to see the same counter.
- Lua script = atomic. Check + decrement in a single Lua script. No race condition.
- Token bucket default. Tokens refill at rate R, each request costs 1. Best default for most APIs.
- Fail behavior. If Redis is down: fail open (allow all) or fail closed (deny all). Name this explicitly.
Tradeoffs
Deep dives
use fixed window counting — INCR a key, expire at the window boundary
the algorithm choice has real operational tradeoffs, not just theoretical ones. Fixed window: simplest (INCR key, expire at window boundary), but the "double spend" attack — a burst of N requests at window T-1 and N requests at window T+1 = 2N requests in a 2-second window while the limit is N per window. Sliding window counter: curr_window_count + (prev_window_count × fraction_of_prev_window_elapsed) ≈ accurate sliding window at low memory cost. Token bucket: refill rate R tokens per second, cost 1 per request, bucket size B allows bursts up to B. Leaky bucket: queue requests and process at fixed rate — smoothest, but adds latency
sliding window counter for most API rate limiting (accurate, cheap). Token bucket for APIs that explicitly want to allow short bursts (e.g., batch endpoints). Fixed window only for very simple use cases where boundary bursts are acceptable. The algorithm choice should be justified against your NFRs — don't just name it
The race condition: two servers both read the same counter (value: 999), both check 999 < 1000 (pass), both increment to 1000. User gets 2× their quota. The naive GET → check → INCR is broken
use Redis transactions (MULTI/EXEC)
Redis Lua scripts — the script executes atomically on the Redis server; no other command can run between any two lines of the Lua script. The script: (1) LOLWUT or EXISTS to check the key, (2) GET the current count, (3) if count < limit, INCR and return allowed, else return denied. All as one atomic operation
why not MULTI/EXEC? MULTI/EXEC is optimistic — if another client modifies the key between WATCH and EXEC, the transaction fails and must retry. Under high concurrency this creates a hot retry loop. Lua is unconditional — no retry needed. INCR alone is atomic too, but only for the increment — the conditional check requires Lua
fail closed — return 429 to all requests when Redis is down
what happens when Redis is unavailable? This is the question that separates senior from staff. Options: (1) fail open — allow all requests. Better for user-facing APIs where availability > protection. (2) Fail closed — deny all requests with 429. Better for auth endpoints where one compromised request is worse than a temporary outage. (3) Local in-process fallback — each app server maintains its own token bucket. Allows N × (num_servers) total requests instead of N, but prevents total failure
design: the failure mode should be a configuration option, not a code decision. Different endpoints have different failure mode requirements. The rate limiter must be explicit about which mode it's in and expose this via metrics. Additionally: Redis Sentinel for HA (automatic failover in <30s); circuit breaker pattern at the rate limiter client so a slow Redis doesn't add latency to every API request
Why the deep dives connect to the scaling problem: "Fast per-request shared counter decisions." Deep dive 1 solves algorithm correctness. Deep dive 2 solves distributed atomicity. Deep dive 3 solves failure handling.
Interview script
Whiteboard
+----------------------+
| Config Service |
| rules and limits |
+----------+-----------+
|
periodic sync
|
+---------+ HTTPS +------------v-------------+
| Clients | --------------> | API Gateway / LB Layer |
| users | | auth parse + rate limit |
| IPs | | check before app traffic |
+---------+ +------+---------+---------+
| |
allow request | | reject request
| |
| v
| +----------------------+
| | 429 Response Builder |
| | limit remaining reset|
| +----------------------+
|
v
+---------+----------+
| Backend Services |
| social media APIs |
+--------------------+
Inside the gateway rate limiter path
extract client key
userId or IP or apiKey + endpoint rule
|
v
+----------+-----------+
| Shard Router |
| hash client key |
+----------+-----------+
|
v
+----------+-----------------------------------+
| Redis Cluster |
| shared bucket state across gateway instances |
| |
| shard 1 shard 2 shard 3 ... |
| +--------+ +--------+ +--------+ |
| |alice | |bob | |carol | |
| |tokens | |tokens | |tokens | |
| |refill | |refill | |refill | |
| +--------+ +--------+ +--------+ |
+-------------------+--------------------------+
|
atomic Lua script
|
v
read bucket -> refill tokens -> consume 1 -> return decision
Per shard HA
+------------------+
| Redis Primary |
+--------+---------+
|
replicate
|
+--------v---------+
| Redis Replica |
+------------------+
Request flowvideo_id
100ms batch window
A real-time fan conversation around a live video with potentially 1M+ simultaneous viewers.
The hard part: naive per-comment fan-out to 1M SSE connections would collapse the system.
The hard part is fan-out under extreme skew. One new comment is a tiny write, but it may need to reach thousands or millions of viewers for the same live video almost immediately.
That creates three scaling pain points. First, the read side explodes because every active viewer needs a near real-time stream, so polling falls over and even push connections become expensive at big scale. Second, hot videos create uneven load. Most streams are quiet, but one viral stream can concentrate huge traffic, connection count, and comment throughput onto a small part of the system. Third, once viewers for the same video are spread across many realtime servers, you need coordination so every server knows which new comments to forward.
The extra twist is that the best design changes for mega-streams. For normal videos, SSE plus pub sub works well. For massive streams, you usually stop trying to show every comment and switch to sampling or snapshot style delivery because humans cannot read thousands of comments per second anyway.
- Scope it first. Core: post comment, deliver to all live viewers in near-real-time, show recent comment history on join. Out of scope: reactions, moderation pipeline, comment replies.
- Batch — the single most important decision. 100ms coalescing window. At 1M viewers and 1K comments/sec, per-comment push = 1B SSE pushes/sec. Batching reduces this to 10M/sec. This is a requirement, not an optimization.
- Kafka partition = delivery server locality. Partition key = video_id. One partition consumed by one delivery server group. All viewer connections for a video are on those servers. Zero cross-server fan-out coordination needed.
- SSE over WebSocket. Comments flow one direction: server → viewer. SSE is simpler (standard HTTP/2, auto-reconnect, no upgrade). Only use WebSocket if viewers need to send data inline.
- Late joiner catch-up. On connect, client sends last_seen_comment_id. Server queries Cassandra for comments after that ID (LIMIT 30), then switches to live SSE. Client deduplicates by comment_id.
- Sampling at extreme scale. 50M viewers (Super Bowl): show 0.1% of comments — still 1 comment/sec per viewer at 1K comments/sec. Humans cannot read faster. VIP comments (verified accounts) always shown regardless of sampling rate.
- Failure mode to name. Delivery server crash: client auto-reconnects, sends last_seen_comment_id, replays missed comments from Cassandra. Cassandra is the durable fallback — never rely on SSE delivery alone for correctness.
1M viewers × 1 comment per 100ms = 10M SSE pushes per second if done naively. No system survives this. The critical insight: batching is not an optimization, it's a requirement
batch at the delivery server
batch at the Kafka consumption layer. Delivery server consumes Kafka for a video_id partition, buffers incoming comment events for 100ms, then pushes a single batch payload to all 10K connected clients. The batch payload: {comments: [{id, text, author, timestamp}, ...], viewer_count: 1000000}
the batching window is a tunable parameter. 100ms gives 10 batches/second — fast enough for a live conversation feel, slow enough to be viable at scale. For ultra-high-volume streams: adaptive batching — increase window size as comment rate increases, keeping per-viewer bandwidth constant regardless of comment volume. The coalescing also improves compression ratio on the SSE payload (repeated fields in multiple comments compress well)
route comment events to any available delivery server, then fan-out via cross-server pub/sub
Kafka partition key = video_id. All comments for a video go to one Kafka partition. One delivery server group (2-3 servers) consumes each partition. By contract, all viewers of a video are connected to servers in the group that consumes that video's partition. Result: fan-out requires zero cross-server communication — the consuming server directly has all viewer connections
what about load balancing? A viral video creates a hot partition (1M viewers on 2-3 servers). Mitigation: hot video detection — when a video exceeds a viewer threshold, allocate multiple Kafka partitions for it and a larger server group. Connection routing: clients are directed to servers in the right group via DNS-based load balancing with video_id affinity. For mega-streams (Super Bowl: 50M+ viewers): dedicated server cluster for the stream, isolated from regular traffic, pre-provisioned
send the entire comment history on connect, then switch to live stream
a viewer joins a live stream 30 minutes in. They need: (1) the last N comments to provide context, (2) then a seamless transition to the live comment stream. Weak answer: load from DB then connect to SSE. Strong answer: explicit catch-up protocol. On SSE connection: client sends last_comment_id = 0 (new viewer). Server queries Cassandra: SELECT * FROM comments WHERE video_id=? AND id > 0 ORDER BY id ASC LIMIT 30. Returns the 30 most recent comments. Then switches client to live SSE stream. Client deduplicates by comment_id in case the SSE stream delivers a comment that was already in the catch-up response
the catch-up query uses a cursor (comment_id) not a timestamp — comment_ids are monotonically increasing Snowflake IDs that encode time, so they're both ordered and unique. The transition from catch-up to live is seamless: client tracks max comment_id received, SSE stream starts from next ID
Why the deep dives connect to the scaling problem: "Fan-out under extreme skew." Deep dive 1 solves the fan-out rate problem. Deep dive 2 solves the routing problem. Deep dive 3 solves the late-joiner problem.
+-------------------+
| Commenter App |
| POST comment |
+---------+---------+
|
v
+---------------------+
| API / Comment |
| Management Service |
+----+------------+---+
| |
write comment| | publish event
v v
+----------------+ +-------------------+
| Comments DB | | Pub/Sub Bus |
| DynamoDB | | Redis or similar |
+----------------+ +---------+---------+
|
fan out to interested servers
|
+-------------------------+-------------------------+
| | |
v v v
+---------------+ +---------------+ +---------------+
| Realtime Srv 1| | Realtime Srv 2| ... | Realtime Srv N|
| SSE conns | | SSE conns | | SSE conns |
| local map | | local map | | local map |
+------+--------+ +------+--------+ +------+--------+
| | |
v v v
+-------------+ +-------------+ +-------------+
| Viewer Apps | | Viewer Apps | | Viewer Apps |
| SSE stream | | SSE stream | | SSE stream |
+-------------+ +-------------+ +-------------+
History and catch-up path
Viewer App
|
| GET /comments/:liveVideoId?cursor=lastCommentId&pageSize=10
v
+-------------------+
| Comment Management |
| Service |
+---------+---------+
|
v
+-------------------+
| Comments DB |
| paginated reads |
+-------------------+If you want the best interview version, I would say it out loud like this. Comments are written through a comment service into DynamoDB. New comment events are published to a pub sub layer. Realtime servers hold SSE connections to viewers and push comments out. Historical comments and reconnect catch-up come from the database using cursor pagination.
For scale, you'll want one extra note on the diagram. Put a load balancer in front of the realtime servers and say you try to co-locate viewers of the same liveVideoId on the same server or small set of servers to reduce fanout waste.
Problem
A real-time fan conversation around a live video with potentially 1M+ simultaneous viewers.
The hard part: naive per-comment fan-out to 1M SSE connections would collapse the system.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is fan-out under extreme skew. One new comment is a tiny write, but it may need to reach thousands or millions of viewers for the same live video almost immediately.
That creates three scaling pain points. First, the read side explodes because every active viewer needs a near real-time stream, so polling falls over and even push connections become expensive at big scale. Second, hot videos create uneven load. Most streams are quiet, but one viral stream can concentrate huge traffic, connection count, and comment throughput onto a small part of the system. Third, once viewers for the same video are spread across many realtime servers, you need coordination so every server knows which new comments to forward.
The extra twist is that the best design changes for mega-streams. For normal videos, SSE plus pub sub works well. For massive streams, you usually stop trying to show every comment and switch to sampling or snapshot style delivery because humans cannot read thousands of comments per second anyway.
Key points
- Scope it first. Core: post comment, deliver to all live viewers in near-real-time, show recent comment history on join. Out of scope: reactions, moderation pipeline, comment replies.
- Batch — the single most important decision. 100ms coalescing window. At 1M viewers and 1K comments/sec, per-comment push = 1B SSE pushes/sec. Batching reduces this to 10M/sec. This is a requirement, not an optimization.
- Kafka partition = delivery server locality. Partition key = video_id. One partition consumed by one delivery server group. All viewer connections for a video are on those servers. Zero cross-server fan-out coordination needed.
- SSE over WebSocket. Comments flow one direction: server → viewer. SSE is simpler (standard HTTP/2, auto-reconnect, no upgrade). Only use WebSocket if viewers need to send data inline.
- Late joiner catch-up. On connect, client sends last_seen_comment_id. Server queries Cassandra for comments after that ID (LIMIT 30), then switches to live SSE. Client deduplicates by comment_id.
- Sampling at extreme scale. 50M viewers (Super Bowl): show 0.1% of comments — still 1 comment/sec per viewer at 1K comments/sec. Humans cannot read faster. VIP comments (verified accounts) always shown regardless of sampling rate.
- Failure mode to name. Delivery server crash: client auto-reconnects, sends last_seen_comment_id, replays missed comments from Cassandra. Cassandra is the durable fallback — never rely on SSE delivery alone for correctness.
Tradeoffs
Deep dives
1M viewers × 1 comment per 100ms = 10M SSE pushes per second if done naively. No system survives this. The critical insight: batching is not an optimization, it's a requirement
batch at the delivery server
batch at the Kafka consumption layer. Delivery server consumes Kafka for a video_id partition, buffers incoming comment events for 100ms, then pushes a single batch payload to all 10K connected clients. The batch payload: {comments: [{id, text, author, timestamp}, ...], viewer_count: 1000000}
the batching window is a tunable parameter. 100ms gives 10 batches/second — fast enough for a live conversation feel, slow enough to be viable at scale. For ultra-high-volume streams: adaptive batching — increase window size as comment rate increases, keeping per-viewer bandwidth constant regardless of comment volume. The coalescing also improves compression ratio on the SSE payload (repeated fields in multiple comments compress well)
route comment events to any available delivery server, then fan-out via cross-server pub/sub
Kafka partition key = video_id. All comments for a video go to one Kafka partition. One delivery server group (2-3 servers) consumes each partition. By contract, all viewers of a video are connected to servers in the group that consumes that video's partition. Result: fan-out requires zero cross-server communication — the consuming server directly has all viewer connections
what about load balancing? A viral video creates a hot partition (1M viewers on 2-3 servers). Mitigation: hot video detection — when a video exceeds a viewer threshold, allocate multiple Kafka partitions for it and a larger server group. Connection routing: clients are directed to servers in the right group via DNS-based load balancing with video_id affinity. For mega-streams (Super Bowl: 50M+ viewers): dedicated server cluster for the stream, isolated from regular traffic, pre-provisioned
send the entire comment history on connect, then switch to live stream
a viewer joins a live stream 30 minutes in. They need: (1) the last N comments to provide context, (2) then a seamless transition to the live comment stream. Weak answer: load from DB then connect to SSE. Strong answer: explicit catch-up protocol. On SSE connection: client sends last_comment_id = 0 (new viewer). Server queries Cassandra: SELECT * FROM comments WHERE video_id=? AND id > 0 ORDER BY id ASC LIMIT 30. Returns the 30 most recent comments. Then switches client to live SSE stream. Client deduplicates by comment_id in case the SSE stream delivers a comment that was already in the catch-up response
the catch-up query uses a cursor (comment_id) not a timestamp — comment_ids are monotonically increasing Snowflake IDs that encode time, so they're both ordered and unique. The transition from catch-up to live is seamless: client tracks max comment_id received, SSE stream starts from next ID
Why the deep dives connect to the scaling problem: "Fan-out under extreme skew." Deep dive 1 solves the fan-out rate problem. Deep dive 2 solves the routing problem. Deep dive 3 solves the late-joiner problem.
Interview script
Whiteboard
+-------------------+
| Commenter App |
| POST comment |
+---------+---------+
|
v
+---------------------+
| API / Comment |
| Management Service |
+----+------------+---+
| |
write comment| | publish event
v v
+----------------+ +-------------------+
| Comments DB | | Pub/Sub Bus |
| DynamoDB | | Redis or similar |
+----------------+ +---------+---------+
|
fan out to interested servers
|
+-------------------------+-------------------------+
| | |
v v v
+---------------+ +---------------+ +---------------+
| Realtime Srv 1| | Realtime Srv 2| ... | Realtime Srv N|
| SSE conns | | SSE conns | | SSE conns |
| local map | | local map | | local map |
+------+--------+ +------+--------+ +------+--------+
| | |
v v v
+-------------+ +-------------+ +-------------+
| Viewer Apps | | Viewer Apps | | Viewer Apps |
| SSE stream | | SSE stream | | SSE stream |
+-------------+ +-------------+ +-------------+
History and catch-up path
Viewer App
|
| GET /comments/:liveVideoId?cursor=lastCommentId&pageSize=10
v
+-------------------+
| Comment Management |
| Service |
+---------+---------+
|
v
+-------------------+
| Comments DB |
| paginated reads |
+-------------------+If you want the best interview version, I would say it out loud like this. Comments are written through a comment service into DynamoDB. New comment events are published to a pub sub layer. Realtime servers hold SSE connections to viewers and push comments out. Historical comments and reconnect catch-up come from the database using cursor pagination.
For scale, you'll want one extra note on the diagram. Put a load balancer in front of the realtime servers and say you try to co-locate viewers of the same liveVideoId on the same server or small set of servers to reduce fanout waste.
+privacy_level
privacy filter
post-query
privacy_level field. On search, ES handles text matching. A post-query app layer filter enforces friend-list visibility — friend lists are too large and dynamic for ES.Keyword search over billions of posts at Facebook scale.
You can't scan raw post text at request time — the corpus is enormous. The real problem is precomputing an inverted index and keeping it fresh.
The hard part in FB Post Search is that search looks read heavy, but the indexing work is actually very write heavy. Every new post fans out into many index updates, and likes can create even more updates if you sort by popularity.
There are three scaling pain points to call out. First, you cannot scan raw posts at query time because the data is far too large, so you need an inverted index that maps terms to post IDs. Second, some terms are extremely hot, which means their posting lists get huge and expensive to store, sort, and query. Third, freshness matters. New posts should appear quickly, so you need fast ingestion and index updates without overwhelming the system.
A fourth issue is sorting. Recency is manageable, but sorting by like count is much harder because likes change constantly. That means one user action can force many index updates unless you use approximation or batching.
So the short interview answer is this. FB Post Search is hard because it combines massive inverted indexes, hot keywords, heavy write amplification from posts and likes, and a tight freshness requirement.
- Inverted index. Each keyword points to post IDs that contain it. Never scan raw posts at search time.
- Write path. Post → Kafka → index worker tokenizes → updates inverted index in ES.
- Two sort indexes. One ordered by creation time (recency). One ordered by like count (popularity).
- Privacy in app layer. Post-query filter. Friend lists are too large and dynamic for ES.
- Shard by keyword. So writes and reads spread across many machines.
one unified Elasticsearch index, sort at query time
the core data structure: a map from each keyword to an ordered list of post IDs containing that keyword. Two sort orders are required: recency (post_id DESC or created_at DESC) and popularity (like_count DESC). Weak answer: one index, sort at query time. Strong answer: two separate indexes, each optimized for its sort order. The recency index is append-only (new posts are always the most recent — just append to the front). The like-count index is updated on every like — much more expensive because one like on a popular post triggers updates to potentially thousands of term lists
don't update the like-count index on every individual like. Batch like updates in Redis: increment a like counter in Redis, flush to the index every 5 minutes. This reduces write amplification from O(terms_per_post) per like to O(terms_per_post) per 5-minute window. The tradeoff: like-count rankings are slightly stale (up to 5 minutes) — acceptable, as users don't notice
Privacy is the hardest part of FB Post Search. Every result must be filtered against the viewer's permission to see it
post-query filter for every result
coarse filter in ES (privacy_level field: PUBLIC, FRIENDS, PRIVATE) combined with a friend-graph check in the app layer for FRIENDS-only posts
why not store friend lists in ES documents? The friend list for a user can be millions of people and changes constantly — storing it in every post document would make ES documents enormous and constantly out of date. The hybrid approach: ES returns candidates filtered by privacy_level. For FRIENDS posts in the result set, batch-query the social graph service: "Is viewer X friends with any of these [user_ids]?" This is a set intersection problem — solved efficiently with Bloom filters per user (is viewer in author's friend set?) or with a graph adjacency lookup. The friend check adds ~20ms to P99 latency — acceptable for a search query
sync Elasticsearch from PostgreSQL with a scheduled batch job every 10 minutes
users expect to search for something they just posted and find it. The ingestion pipeline: post created in PG → Kafka event → index worker → ES update. End-to-end target: < 1 minute
bottlenecks to address: (1) ES bulk indexing batch size — larger batches are more efficient but add latency. At 5,787 posts/second, even 1-second batches give reasonable throughput. (2) ES refresh interval — by default ES refreshes the search index every 1 second (making indexed docs visible to search). For breaking news content: reduce refresh interval to 100ms for the first 5 minutes after a high-engagement post is created (dynamic refresh rate based on post engagement velocity). (3) High-engagement posts fast-lane: ML classifier identifies potentially viral posts within seconds of publish — route these to a priority Kafka topic with a dedicated low-latency indexer
Why the deep dives connect to the scaling problem: "Massive inverted indexes, hot keywords, write amplification, and freshness." Each deep dive addresses one constraint.
Two-path script.
+-------------------+
User Search Request -----> | CDN / Edge |
+---------+---------+
|
v
+-------------------+
| API Gateway |
| auth rate limit |
+---------+---------+
|
v
+-------------------+
| Search Service |
+----+----------+---+
| |
cache hit? | | fetch posts and fresh likes
| v
| +--------------+
| | Post Service |
| +--------------+
| +--------------+
| | Like Service |
| +--------------+
v
+-----------------------+
| Distributed Search |
| Cache TTL < 1 minute |
+-----------+-----------+
|
cache miss
|
v
+-------------------------------------------+
| Keyword Index Store |
| sharded by keyword |
| |
| creation index = list by recency |
| likes index = sorted set by like score |
+-------------------+-----------------------+
|
hot keywords | cold keywords
|
+---------------+---------------+
| |
v v
+-------------+ +-------------+
| Redis shard | | Blob store |
| in memory | | S3 or R2 |
+-------------+ +-------------+
Write path
==========
Post Create ----> Post Service ----+
|
Like Event -----> Like Service ----+----> Kafka or event log ----> Ingestion workers
|
v
+----------------------+
| Tokenizer |
| split into keywords |
| optional bigrams |
+----------+-----------+
|
+------------------------+----------------------+
| |
v v
+--------------------------+ +--------------------------+
| Update creation indexes | | Update likes indexes |
| add postId per keyword | | sorted set score updates |
+--------------------------+ +--------------------------+
|
v
+------------------------------+
| Optional like batcher or |
| approximate milestone writer |
+------------------------------+The mental model is two pipelines. One pipeline builds keyword indexes from posts and likes. The other pipeline serves search by reading those indexes fast, usually from cache or Redis.
If you were drawing this in an interview, I would start with just User, API Gateway, Search Service, Ingestion Service, and Index Store. Then add cache, Kafka, like batching, and cold storage only if the interviewer pushes on scale or freshness trade-offs.
Problem
Keyword search over billions of posts at Facebook scale.
You can't scan raw post text at request time — the corpus is enormous. The real problem is precomputing an inverted index and keeping it fresh.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in FB Post Search is that search looks read heavy, but the indexing work is actually very write heavy. Every new post fans out into many index updates, and likes can create even more updates if you sort by popularity.
There are three scaling pain points to call out. First, you cannot scan raw posts at query time because the data is far too large, so you need an inverted index that maps terms to post IDs. Second, some terms are extremely hot, which means their posting lists get huge and expensive to store, sort, and query. Third, freshness matters. New posts should appear quickly, so you need fast ingestion and index updates without overwhelming the system.
A fourth issue is sorting. Recency is manageable, but sorting by like count is much harder because likes change constantly. That means one user action can force many index updates unless you use approximation or batching.
So the short interview answer is this. FB Post Search is hard because it combines massive inverted indexes, hot keywords, heavy write amplification from posts and likes, and a tight freshness requirement.
Key points
- Inverted index. Each keyword points to post IDs that contain it. Never scan raw posts at search time.
- Write path. Post → Kafka → index worker tokenizes → updates inverted index in ES.
- Two sort indexes. One ordered by creation time (recency). One ordered by like count (popularity).
- Privacy in app layer. Post-query filter. Friend lists are too large and dynamic for ES.
- Shard by keyword. So writes and reads spread across many machines.
Tradeoffs
Deep dives
one unified Elasticsearch index, sort at query time
the core data structure: a map from each keyword to an ordered list of post IDs containing that keyword. Two sort orders are required: recency (post_id DESC or created_at DESC) and popularity (like_count DESC). Weak answer: one index, sort at query time. Strong answer: two separate indexes, each optimized for its sort order. The recency index is append-only (new posts are always the most recent — just append to the front). The like-count index is updated on every like — much more expensive because one like on a popular post triggers updates to potentially thousands of term lists
don't update the like-count index on every individual like. Batch like updates in Redis: increment a like counter in Redis, flush to the index every 5 minutes. This reduces write amplification from O(terms_per_post) per like to O(terms_per_post) per 5-minute window. The tradeoff: like-count rankings are slightly stale (up to 5 minutes) — acceptable, as users don't notice
Privacy is the hardest part of FB Post Search. Every result must be filtered against the viewer's permission to see it
post-query filter for every result
coarse filter in ES (privacy_level field: PUBLIC, FRIENDS, PRIVATE) combined with a friend-graph check in the app layer for FRIENDS-only posts
why not store friend lists in ES documents? The friend list for a user can be millions of people and changes constantly — storing it in every post document would make ES documents enormous and constantly out of date. The hybrid approach: ES returns candidates filtered by privacy_level. For FRIENDS posts in the result set, batch-query the social graph service: "Is viewer X friends with any of these [user_ids]?" This is a set intersection problem — solved efficiently with Bloom filters per user (is viewer in author's friend set?) or with a graph adjacency lookup. The friend check adds ~20ms to P99 latency — acceptable for a search query
sync Elasticsearch from PostgreSQL with a scheduled batch job every 10 minutes
users expect to search for something they just posted and find it. The ingestion pipeline: post created in PG → Kafka event → index worker → ES update. End-to-end target: < 1 minute
bottlenecks to address: (1) ES bulk indexing batch size — larger batches are more efficient but add latency. At 5,787 posts/second, even 1-second batches give reasonable throughput. (2) ES refresh interval — by default ES refreshes the search index every 1 second (making indexed docs visible to search). For breaking news content: reduce refresh interval to 100ms for the first 5 minutes after a high-engagement post is created (dynamic refresh rate based on post engagement velocity). (3) High-engagement posts fast-lane: ML classifier identifies potentially viral posts within seconds of publish — route these to a priority Kafka topic with a dedicated low-latency indexer
Why the deep dives connect to the scaling problem: "Massive inverted indexes, hot keywords, write amplification, and freshness." Each deep dive addresses one constraint.
Interview script
Two-path script.
Whiteboard
+-------------------+
User Search Request -----> | CDN / Edge |
+---------+---------+
|
v
+-------------------+
| API Gateway |
| auth rate limit |
+---------+---------+
|
v
+-------------------+
| Search Service |
+----+----------+---+
| |
cache hit? | | fetch posts and fresh likes
| v
| +--------------+
| | Post Service |
| +--------------+
| +--------------+
| | Like Service |
| +--------------+
v
+-----------------------+
| Distributed Search |
| Cache TTL < 1 minute |
+-----------+-----------+
|
cache miss
|
v
+-------------------------------------------+
| Keyword Index Store |
| sharded by keyword |
| |
| creation index = list by recency |
| likes index = sorted set by like score |
+-------------------+-----------------------+
|
hot keywords | cold keywords
|
+---------------+---------------+
| |
v v
+-------------+ +-------------+
| Redis shard | | Blob store |
| in memory | | S3 or R2 |
+-------------+ +-------------+
Write path
==========
Post Create ----> Post Service ----+
|
Like Event -----> Like Service ----+----> Kafka or event log ----> Ingestion workers
|
v
+----------------------+
| Tokenizer |
| split into keywords |
| optional bigrams |
+----------+-----------+
|
+------------------------+----------------------+
| |
v v
+--------------------------+ +--------------------------+
| Update creation indexes | | Update likes indexes |
| add postId per keyword | | sorted set score updates |
+--------------------------+ +--------------------------+
|
v
+------------------------------+
| Optional like batcher or |
| approximate milestone writer |
+------------------------------+The mental model is two pipelines. One pipeline builds keyword indexes from posts and likes. The other pipeline serves search by reading those indexes fast, usually from cache or Redis.
If you were drawing this in an interview, I would start with just User, API Gateway, Search Service, Ingestion Service, and Index Store. Then add cache, Kafka, like batching, and cold storage only if the interviewer pushes on scale or freshness trade-offs.
+ full-text
Local business discovery and trust. Find a place nearby, filter by category and location, decide based on reviews and ratings.
Two challenges: geospatial search at scale, and keeping ratings consistent.
The hard part in Yelp is search, not writes. You need to search by text, category, and location at the same time, and location is especially tricky because geospatial queries do not scale well with a plain relational lookup.
There are three main pain points you should call out. First, search is multi-dimensional. A user might ask for coffee, in a certain area, in a certain category, so you need full-text indexing plus geospatial indexing, not just a normal database query. Second, reads dominate writes by a lot, so popular areas create heavy search and business page traffic and you need read scaling with replicas or caching. Third, some derived data like average rating must stay cheap to read, so you usually precompute it instead of recalculating from all reviews on every search.
A good interview summary is this. Yelp is hard because it combines read-heavy traffic with geospatial search and text search in one request. The write path for reviews is comparatively small, so the main challenge is making search fast while keeping results fresh enough.
- PostgreSQL = source of truth. Businesses, reviews, and users.
- Elasticsearch = search index. geo_distance filter + full-text in one query.
- Redis = rating cache. Average ratings served from Redis for fast reads. Synced to PG asynchronously.
- Photos = S3 + CDN. Never serve photos from app servers.
- Search → ES, not PG. Never do SQL LIKE queries on business names at scale.
run separate queries for geo (PostGIS) and text (LIKE), merge results in the app layer
the scaling pain is that a user query like "best sushi near me open now" combines full-text search, geo filtering, category filtering, and rating sorting in one request. Weak answer: separate queries to separate services, merge in app layer. Strong answer: Elasticsearch as a single query surface with a compound query: geo_distance filter (within X km of lat/lng) + multi_match on business name/category/description + term filter on category + range filter on rating + sort
the ES query shape matters for performance. Structured fields (category, rating, hours) should use filter context (cached, no scoring overhead) and text fields should use query context (scored, more expensive). The geo_distance filter is the most selective first — apply it first to reduce the candidate set before text scoring. For "open now": this is a computed field that changes every minute — too dynamic to index. Apply as a post-query filter in app layer, not in ES. Cache "is_open" status per business with 15-minute TTL
ES is derived data — PostgreSQL is the source of truth. Sync strategy options: (1) dual write (write to PG + ES in same request) — simple but PG and ES can diverge on partial failure; (2) CDC (Debezium reads PG WAL → Kafka → ES indexer) — eventual consistency, reliable, standard production approach; (3) async event-driven (after PG write succeeds, publish Kafka event → ES indexer) — same reliability as CDC but more explicit
Oversimplify keeping es in sync with postgresql — name one component, skip failure modes and metrics.
ES is derived data — PostgreSQL is the source of truth. Sync strategy options: (1) dual write (write to PG + ES in same request) — simple but PG and ES can diverge on partial failure; (2) CDC (Debezium reads PG WAL → Kafka → ES indexer) — eventual consistency, reliable, standard production approach; (3) async event-driven (after PG write succeeds, publish Kafka event → ES indexer) — same reliability as CDC but more explicit
CDC is the production default because it works for all write patterns (including bulk imports, migrations, and direct DB writes by other services) without requiring every writer to know about ES. Lag: typically 1-5 seconds — acceptable for search. Monitor lag metric: alert at >60s. For new business data (most time-sensitive): reduce batch size in the indexer for faster propagation
The access pattern for ratings: millions of reads per business page, one write per new review. Recomputing avg(rating) from all reviews on every page view is O(N) per request at 11,600 QPS — impossible
cache the result
maintain a running average in the database: (sum_of_ratings, review_count) per business. On new review: UPDATE businesses SET sum_of_ratings = sum_of_ratings + ?, review_count = review_count + 1 WHERE id = ?. Average = sum/count, computed at read time (O(1))
concurrent reviews on a popular business can create a write hotspot on the businesses row. Mitigation: use PG row-level locking for the increment (short lock duration), or batch review aggregation (flush accumulated ratings every 30s from Redis). In ES: rating field updated via the same CDC pipeline — search results always reflect recent ratings
Why the deep dives connect to the scaling problem: "Multi-dimensional search plus precomputed read data." Each deep dive solves one dimension of the read scaling problem.
+-------------------+
| Web / Mobile |
| Client |
+---------+---------+
|
v
+-------------------+
| API Gateway |
+----+---------+----+
| |
GET search / view | POST review
| |
v v
+----------------+ +----------------+
| Business | | Review |
| Service | | Service |
+---+--------+---+ +---+--------+---+
| | | |
| | | |
| | | +------------------+
| | | |
| | v v
| | +------------------+ +------------------+
| | | Reviews Table | | Businesses Table |
| | | unique(userId, | | avg_rating |
| | | businessId) | | num_reviews |
| | +---------+--------+ +---------+--------+
| | | ^
| +-------------+--------------------------|
| sync rating update |
| optimistic locking on write |
| |
v |
+-------------------------+ |
| Read Replica / Cache |----------------------------------+
| for hot business reads |
+-----------+-------------+
|
v
+-------------------------+
| Search Store |
| Elasticsearch or |
| Postgres + PostGIS |
| + full text indexes |
+-----------+-------------+
^
|
CDC / async indexing
|
+-----------+-------------+
| Primary DB |
| businesses + reviews |
+-------------------------+
Optional for named locations
+-------------------------+
| Locations Table |
| name -> polygon |
| city / neighborhood |
+-------------------------+The main story you should tell is simple. Search and business reads go through the Business Service, reviews go through the Review Service, the primary database is the source of truth, and search is powered either by Elasticsearch with CDC sync or by Postgres extensions if you want the simpler version. For interviews, I would present the simple version first, then add the search store and location polygons only if the interviewer pushes on search quality or scale.
Problem
Local business discovery and trust. Find a place nearby, filter by category and location, decide based on reviews and ratings.
Two challenges: geospatial search at scale, and keeping ratings consistent.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in Yelp is search, not writes. You need to search by text, category, and location at the same time, and location is especially tricky because geospatial queries do not scale well with a plain relational lookup.
There are three main pain points you should call out. First, search is multi-dimensional. A user might ask for coffee, in a certain area, in a certain category, so you need full-text indexing plus geospatial indexing, not just a normal database query. Second, reads dominate writes by a lot, so popular areas create heavy search and business page traffic and you need read scaling with replicas or caching. Third, some derived data like average rating must stay cheap to read, so you usually precompute it instead of recalculating from all reviews on every search.
A good interview summary is this. Yelp is hard because it combines read-heavy traffic with geospatial search and text search in one request. The write path for reviews is comparatively small, so the main challenge is making search fast while keeping results fresh enough.
Key points
- PostgreSQL = source of truth. Businesses, reviews, and users.
- Elasticsearch = search index. geo_distance filter + full-text in one query.
- Redis = rating cache. Average ratings served from Redis for fast reads. Synced to PG asynchronously.
- Photos = S3 + CDN. Never serve photos from app servers.
- Search → ES, not PG. Never do SQL LIKE queries on business names at scale.
Tradeoffs
Deep dives
run separate queries for geo (PostGIS) and text (LIKE), merge results in the app layer
the scaling pain is that a user query like "best sushi near me open now" combines full-text search, geo filtering, category filtering, and rating sorting in one request. Weak answer: separate queries to separate services, merge in app layer. Strong answer: Elasticsearch as a single query surface with a compound query: geo_distance filter (within X km of lat/lng) + multi_match on business name/category/description + term filter on category + range filter on rating + sort
the ES query shape matters for performance. Structured fields (category, rating, hours) should use filter context (cached, no scoring overhead) and text fields should use query context (scored, more expensive). The geo_distance filter is the most selective first — apply it first to reduce the candidate set before text scoring. For "open now": this is a computed field that changes every minute — too dynamic to index. Apply as a post-query filter in app layer, not in ES. Cache "is_open" status per business with 15-minute TTL
ES is derived data — PostgreSQL is the source of truth. Sync strategy options: (1) dual write (write to PG + ES in same request) — simple but PG and ES can diverge on partial failure; (2) CDC (Debezium reads PG WAL → Kafka → ES indexer) — eventual consistency, reliable, standard production approach; (3) async event-driven (after PG write succeeds, publish Kafka event → ES indexer) — same reliability as CDC but more explicit
Oversimplify keeping es in sync with postgresql — name one component, skip failure modes and metrics.
ES is derived data — PostgreSQL is the source of truth. Sync strategy options: (1) dual write (write to PG + ES in same request) — simple but PG and ES can diverge on partial failure; (2) CDC (Debezium reads PG WAL → Kafka → ES indexer) — eventual consistency, reliable, standard production approach; (3) async event-driven (after PG write succeeds, publish Kafka event → ES indexer) — same reliability as CDC but more explicit
CDC is the production default because it works for all write patterns (including bulk imports, migrations, and direct DB writes by other services) without requiring every writer to know about ES. Lag: typically 1-5 seconds — acceptable for search. Monitor lag metric: alert at >60s. For new business data (most time-sensitive): reduce batch size in the indexer for faster propagation
The access pattern for ratings: millions of reads per business page, one write per new review. Recomputing avg(rating) from all reviews on every page view is O(N) per request at 11,600 QPS — impossible
cache the result
maintain a running average in the database: (sum_of_ratings, review_count) per business. On new review: UPDATE businesses SET sum_of_ratings = sum_of_ratings + ?, review_count = review_count + 1 WHERE id = ?. Average = sum/count, computed at read time (O(1))
concurrent reviews on a popular business can create a write hotspot on the businesses row. Mitigation: use PG row-level locking for the increment (short lock duration), or batch review aggregation (flush accumulated ratings every 30s from Redis). In ES: rating field updated via the same CDC pipeline — search results always reflect recent ratings
Why the deep dives connect to the scaling problem: "Multi-dimensional search plus precomputed read data." Each deep dive solves one dimension of the read scaling problem.
Interview script
Whiteboard
+-------------------+
| Web / Mobile |
| Client |
+---------+---------+
|
v
+-------------------+
| API Gateway |
+----+---------+----+
| |
GET search / view | POST review
| |
v v
+----------------+ +----------------+
| Business | | Review |
| Service | | Service |
+---+--------+---+ +---+--------+---+
| | | |
| | | |
| | | +------------------+
| | | |
| | v v
| | +------------------+ +------------------+
| | | Reviews Table | | Businesses Table |
| | | unique(userId, | | avg_rating |
| | | businessId) | | num_reviews |
| | +---------+--------+ +---------+--------+
| | | ^
| +-------------+--------------------------|
| sync rating update |
| optimistic locking on write |
| |
v |
+-------------------------+ |
| Read Replica / Cache |----------------------------------+
| for hot business reads |
+-----------+-------------+
|
v
+-------------------------+
| Search Store |
| Elasticsearch or |
| Postgres + PostGIS |
| + full text indexes |
+-----------+-------------+
^
|
CDC / async indexing
|
+-----------+-------------+
| Primary DB |
| businesses + reviews |
+-------------------------+
Optional for named locations
+-------------------------+
| Locations Table |
| name -> polygon |
| city / neighborhood |
+-------------------------+The main story you should tell is simple. Search and business reads go through the Business Service, reviews go through the Review Service, the primary database is the source of truth, and search is powered either by Elasticsearch with CDC sync or by Postgres extensions if you want the simpler version. For interviews, I would present the simple version first, then add the search store and location polygons only if the interviewer pushes on search quality or scale.
Hausdorff match
Activity tracking, segment performance, and social fitness. Users upload GPS-tracked workouts, get stats, compete on segment leaderboards.
Hard parts: efficient GPS time-series storage, segment matching, and fast leaderboard reads.
The hard part in Strava is not raw request volume. It is that the product mixes offline tracking, large route data, and social reads in one system.
There are three pain points you should call out. First, each activity generates a long stream of GPS points, so storage grows fast and older route data can get expensive. Second, the app must work with weak or no connectivity, which means the client has to buffer data locally and sync later without losing too much progress. Third, if you add live sharing, the system becomes much harder because now you are handling frequent location updates plus many friends reading those updates at the same time.
A good interview summary is this. Strava is hard because the client does a lot of the work, route data is much heavier than normal app metadata, and real time sharing can turn a simple upload system into a continuous update system.
- Scope it first. Core: upload GPS activity, compute stats, match segments, leaderboards, social feed. Out of scope unless asked: live tracking, route planning, training analytics, coaching.
- Upload is async — always. Client sends polyline-encoded GPS track to S3. Upload API acknowledges immediately. Kafka job triggers async processing. Stats and segment matches appear minutes later — that is the expected UX.
- TimescaleDB for GPS time-series. 300K GPS points/sec appended across all users. TimescaleDB: time-partitioned, columnar compression (5× smaller), SQL joins with activity metadata. Never use plain PostgreSQL for this volume.
- Segment matching — Hausdorff distance. Spatial index on segment bounding boxes (R-tree or geohash). For each activity: query segments whose bounding box overlaps. Then run Hausdorff distance check on candidates only. Reduces 1M segments to ~200 candidates per activity.
- Leaderboard — Redis sorted set with trimming. ZADD leaderboard:{segment_id} elapsed_seconds user_id. ZREMRANGEBYRANK after each ZADD to cap at top 10K entries. ZREVRANGE for display. Historical beyond top 10K fetched from PostgreSQL.
- GPS noise handling. Douglas-Peucker simplification or Kalman filter client-side before upload. Server-side: reject points implying speed > 200 km/h. Noisy tracks create false segment efforts — flag for review.
- Failure mode to name. Segment matching job fails mid-activity: idempotent Kafka consumer replays. Track transcode status per (activity_id, segment_id). Retry only failed segments, not the whole activity.
store GPS points as rows in a PostgreSQL table indexed by activity_id
the core constraint is 300K GPS point writes/second (10M users × 5 activities/week × 3,600 points/activity) combined with complex time-range queries for activity display and segment analysis. Weak answer: PostgreSQL with a GPS_points table. Strong answer: TimescaleDB with a hypertable partitioned by (activity_id, time). Time-partitioned storage means: (1) recent data is in memory or SSD, old data on cheaper storage; (2) time-range queries touch only the relevant partitions; (3) TTL/retention policies drop old partitions cheaply (DROP is instant, DELETE is O(N))
design: GPS_points table has (activity_id, timestamp, lat, lng, elevation, heart_rate, power) as a time-series table. The partition key must be on timestamp for range query efficiency. Compression: TimescaleDB columnar compression reduces GPS data to 20% of original size with no query changes. Rollup: pre-aggregate GPS points to 10-second intervals for activities older than 30 days — reduces storage by 10× at the cost of reduced resolution for old activities (acceptable — users rarely scroll old activity GPS at full resolution)
iterate through all 1M segments and run Hausdorff distance on each for every uploaded activity
every activity must be matched against all segments whose bounding box overlaps the activity's bounding box. At 83 activity uploads/second and 1M segments, naive O(activities × segments) is impossible
approach: spatial index on segment bounding boxes (R-tree or geohash grid). For each activity: (1) compute activity bounding box, (2) query spatial index for segments whose bounding box overlaps, (3) run Hausdorff distance check on the reduced candidate set. The spatial index reduces candidates from 1M to ~100-500 per activity. Hausdorff distance: the maximum of the minimum distances between two polylines. Two polylines match if Hausdorff distance < threshold (e.g., 25 meters). GPS noise handling: smooth the activity GPS track with a Kalman filter or Douglas-Peucker simplification before matching — reduces false negatives from noisy data. Parallel matching: each activity's segment matches can be computed independently — embarrassingly parallel across worker nodes, partitioned by activity bounding box geohash
Each segment has a leaderboard: fastest time by athletes who have completed that segment. The access pattern: frequent reads (users checking their ranking), moderate writes (new segment efforts after activity processing)
Oversimplify segment leaderboards — name one component, skip failure modes and metrics.
Each segment has a leaderboard: fastest time by athletes who have completed that segment. The access pattern: frequent reads (users checking their ranking), moderate writes (new segment efforts after activity processing)
design: Redis sorted set per segment_id (key: leaderboard:{segment_id}, score: elapsed_seconds, member: user_id). ZADD adds a new effort; if the user has a previous effort, ZADD with NX flag only adds new members, so first add only — use regular ZADD but also track the user's best time separately to avoid storing worse efforts. On ZADD, also check if user's new time is better than their stored best (ZSCORE lookup, O(log N)), and only update if improved. ZREVRANGE for top-N is O(log N + K). Leaderboard trimming: ZREMRANGEBYRANK removes bottom entries, keeping only top 10K per segment (50K × 10K × 8 bytes = 4 GB total Redis for all segments). Historical rankings beyond top 10K are fetched from PostgreSQL
Why the deep dives connect to the scaling problem: "Offline tracking, large route data, and social reads." Deep dive 1 solves time-series storage. Deep dive 2 solves segment matching at scale. Deep dive 3 solves leaderboard performance.
+----------------------+
| Mobile App |
| start pause stop |
| local GPS tracking |
| local stats display |
| offline buffer |
+----------+-----------+
|
create sync fetch activities
|
v
+----------------------+
| API or LB Layer |
+----------+-----------+
|
v
+----------------------+
| Activity Service |
| activity lifecycle |
| ingest route uploads |
| fetch activity feed |
+-----+-----------+----+
| |
| |
v v
+------------------+ +------------------+
| Activities DB | | Friends DB |
| activity metadata | | user friendships |
| route points | +------------------+
| state log |
+------------------+
|
|
v
+------------------------------+
| optional cache for hot reads |
+------------------------------+
Read flow
Mobile App -> API -> Activity Service -> Activities DB or Friends DB
Write flow for normal offline-first design
Mobile App stores GPS locally during run
Mobile App uploads completed activity in one sync
Activity Service writes metadata and route data to DB
Optional realtime sharing extension
athlete app sends periodic updates every few seconds
|
v
+----------------------+
| Activity Service |
+----------+-----------+
|
v
+----------------------+
| Activities DB |
+----------+-----------+
|
friends poll for latest activity updates
|
v
+----------------------+
| Friends Apps |
+----------------------+The main idea is that the client does most of the live tracking work locally. That is the key simplification here. The backend mainly handles activity creation, final sync, and reads for completed runs. If you want, I can also give you a cleaner interview-style version that fits in 30 seconds on a whiteboard.
Problem
Activity tracking, segment performance, and social fitness. Users upload GPS-tracked workouts, get stats, compete on segment leaderboards.
Hard parts: efficient GPS time-series storage, segment matching, and fast leaderboard reads.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in Strava is not raw request volume. It is that the product mixes offline tracking, large route data, and social reads in one system.
There are three pain points you should call out. First, each activity generates a long stream of GPS points, so storage grows fast and older route data can get expensive. Second, the app must work with weak or no connectivity, which means the client has to buffer data locally and sync later without losing too much progress. Third, if you add live sharing, the system becomes much harder because now you are handling frequent location updates plus many friends reading those updates at the same time.
A good interview summary is this. Strava is hard because the client does a lot of the work, route data is much heavier than normal app metadata, and real time sharing can turn a simple upload system into a continuous update system.
Key points
- Scope it first. Core: upload GPS activity, compute stats, match segments, leaderboards, social feed. Out of scope unless asked: live tracking, route planning, training analytics, coaching.
- Upload is async — always. Client sends polyline-encoded GPS track to S3. Upload API acknowledges immediately. Kafka job triggers async processing. Stats and segment matches appear minutes later — that is the expected UX.
- TimescaleDB for GPS time-series. 300K GPS points/sec appended across all users. TimescaleDB: time-partitioned, columnar compression (5× smaller), SQL joins with activity metadata. Never use plain PostgreSQL for this volume.
- Segment matching — Hausdorff distance. Spatial index on segment bounding boxes (R-tree or geohash). For each activity: query segments whose bounding box overlaps. Then run Hausdorff distance check on candidates only. Reduces 1M segments to ~200 candidates per activity.
- Leaderboard — Redis sorted set with trimming. ZADD leaderboard:{segment_id} elapsed_seconds user_id. ZREMRANGEBYRANK after each ZADD to cap at top 10K entries. ZREVRANGE for display. Historical beyond top 10K fetched from PostgreSQL.
- GPS noise handling. Douglas-Peucker simplification or Kalman filter client-side before upload. Server-side: reject points implying speed > 200 km/h. Noisy tracks create false segment efforts — flag for review.
- Failure mode to name. Segment matching job fails mid-activity: idempotent Kafka consumer replays. Track transcode status per (activity_id, segment_id). Retry only failed segments, not the whole activity.
Tradeoffs
Deep dives
store GPS points as rows in a PostgreSQL table indexed by activity_id
the core constraint is 300K GPS point writes/second (10M users × 5 activities/week × 3,600 points/activity) combined with complex time-range queries for activity display and segment analysis. Weak answer: PostgreSQL with a GPS_points table. Strong answer: TimescaleDB with a hypertable partitioned by (activity_id, time). Time-partitioned storage means: (1) recent data is in memory or SSD, old data on cheaper storage; (2) time-range queries touch only the relevant partitions; (3) TTL/retention policies drop old partitions cheaply (DROP is instant, DELETE is O(N))
design: GPS_points table has (activity_id, timestamp, lat, lng, elevation, heart_rate, power) as a time-series table. The partition key must be on timestamp for range query efficiency. Compression: TimescaleDB columnar compression reduces GPS data to 20% of original size with no query changes. Rollup: pre-aggregate GPS points to 10-second intervals for activities older than 30 days — reduces storage by 10× at the cost of reduced resolution for old activities (acceptable — users rarely scroll old activity GPS at full resolution)
iterate through all 1M segments and run Hausdorff distance on each for every uploaded activity
every activity must be matched against all segments whose bounding box overlaps the activity's bounding box. At 83 activity uploads/second and 1M segments, naive O(activities × segments) is impossible
approach: spatial index on segment bounding boxes (R-tree or geohash grid). For each activity: (1) compute activity bounding box, (2) query spatial index for segments whose bounding box overlaps, (3) run Hausdorff distance check on the reduced candidate set. The spatial index reduces candidates from 1M to ~100-500 per activity. Hausdorff distance: the maximum of the minimum distances between two polylines. Two polylines match if Hausdorff distance < threshold (e.g., 25 meters). GPS noise handling: smooth the activity GPS track with a Kalman filter or Douglas-Peucker simplification before matching — reduces false negatives from noisy data. Parallel matching: each activity's segment matches can be computed independently — embarrassingly parallel across worker nodes, partitioned by activity bounding box geohash
Each segment has a leaderboard: fastest time by athletes who have completed that segment. The access pattern: frequent reads (users checking their ranking), moderate writes (new segment efforts after activity processing)
Oversimplify segment leaderboards — name one component, skip failure modes and metrics.
Each segment has a leaderboard: fastest time by athletes who have completed that segment. The access pattern: frequent reads (users checking their ranking), moderate writes (new segment efforts after activity processing)
design: Redis sorted set per segment_id (key: leaderboard:{segment_id}, score: elapsed_seconds, member: user_id). ZADD adds a new effort; if the user has a previous effort, ZADD with NX flag only adds new members, so first add only — use regular ZADD but also track the user's best time separately to avoid storing worse efforts. On ZADD, also check if user's new time is better than their stored best (ZSCORE lookup, O(log N)), and only update if improved. ZREVRANGE for top-N is O(log N + K). Leaderboard trimming: ZREMRANGEBYRANK removes bottom entries, keeping only top 10K per segment (50K × 10K × 8 bytes = 4 GB total Redis for all segments). Historical rankings beyond top 10K are fetched from PostgreSQL
Why the deep dives connect to the scaling problem: "Offline tracking, large route data, and social reads." Deep dive 1 solves time-series storage. Deep dive 2 solves segment matching at scale. Deep dive 3 solves leaderboard performance.
Interview script
Whiteboard
+----------------------+
| Mobile App |
| start pause stop |
| local GPS tracking |
| local stats display |
| offline buffer |
+----------+-----------+
|
create sync fetch activities
|
v
+----------------------+
| API or LB Layer |
+----------+-----------+
|
v
+----------------------+
| Activity Service |
| activity lifecycle |
| ingest route uploads |
| fetch activity feed |
+-----+-----------+----+
| |
| |
v v
+------------------+ +------------------+
| Activities DB | | Friends DB |
| activity metadata | | user friendships |
| route points | +------------------+
| state log |
+------------------+
|
|
v
+------------------------------+
| optional cache for hot reads |
+------------------------------+
Read flow
Mobile App -> API -> Activity Service -> Activities DB or Friends DB
Write flow for normal offline-first design
Mobile App stores GPS locally during run
Mobile App uploads completed activity in one sync
Activity Service writes metadata and route data to DB
Optional realtime sharing extension
athlete app sends periodic updates every few seconds
|
v
+----------------------+
| Activity Service |
+----------+-----------+
|
v
+----------------------+
| Activities DB |
+----------+-----------+
|
friends poll for latest activity updates
|
v
+----------------------+
| Friends Apps |
+----------------------+The main idea is that the client does most of the live tracking work locally. That is the key simplification here. The backend mainly handles activity creation, final sync, and reads for completed runs. If you want, I can also give you a cleaner interview-style version that fits in 30 seconds on a whiteboard.
Accepting bids correctly at scale. Ensuring only valid higher bids win, no bids are lost, and everyone sees the current highest bid quickly.
The hard part is contention on a tiny piece of shared state. Many users may bid on the same auction at nearly the same time, but only one current highest bid can be correct.
That creates three scaling pain points. First, bid writes are hot and correctness matters, so you need atomic updates or version checks to avoid accepting stale lower bids. Second, auctions get bursty near the end, so one popular item can suddenly get hammered even if overall traffic looks manageable. Third, users expect live updates, which means one accepted bid may need to fan out quickly to many watchers across many servers. So the short interview answer is that Online Auction is hard because it combines hotspot writes, strict bid consistency, and real time fan-out.
- Scope it first. Core: list item, place bid, real-time bid feed, auction close, winner determination. Out of scope unless asked: payments, shipping, dispute resolution, seller ratings.
- Atomic bid check — Redis Lua CAS. GET current_max, compare, SET if higher — all in one Lua script. Single-threaded Redis execution means no two bids can race. Never use GET + SET as separate operations.
- Redis for speed, PostgreSQL for truth. Redis holds current max bid for sub-millisecond reads during live auction. PostgreSQL stores every bid for audit, dispute resolution, and winner determination. Both required.
- Auction close is a state machine. ACTIVE → CLOSING (stop accepting bids) → CLOSED (determine winner). Close job must be idempotent: UPDATE SET status=CLOSED WHERE status=CLOSING. If it runs twice, second is a no-op.
- Anti-sniping extension. Bid placed in last 3 minutes → extend auction by 3 minutes. UPDATE auctions SET ends_at = ends_at + INTERVAL 3 minutes WHERE id=? AND ends_at - NOW() < 3 minutes. Users prefer fair competition over fixed end time.
- Real-time bid feed — SSE with coalescing. 1K watchers × bid storm in last 30s = massive fan-out. 100ms coalescing: push current max bid state, not every individual bid. Watchers see the current price, not a log of every increment.
- Failure mode to name. Redis goes down mid-auction: freeze auction (reject new bids), reconstruct highest bid from PostgreSQL bid history, resume when Redis recovers. SLA: < 30s pause. PostgreSQL is always authoritative.
Two bidders submit simultaneously: A bids $100, B bids $102, both at the exact same millisecond. The system must accept $102 and reject $100
database transaction with SELECT FOR UPDATE
Redis Lua CAS (Compare-And-Set): (1) GET current_highest_bid, (2) if new_bid > current_highest, SET new_bid, return accepted, else return rejected. All atomic in one Lua script
Database transaction with SELECT FOR UPDATE is also correct but adds 5-20ms PG lock latency under high contention. Redis Lua CAS is <1ms. The tradeoff: Redis is not durable by default (RDB snapshot may be seconds old), so every accepted bid must also be written to PostgreSQL before returning success. The two-write pattern: Redis for speed and ordering, PG for durability. If Redis and PG diverge (Redis accepts a bid but PG write fails): on startup/recovery, the PG bid history is authoritative. Redis state is reconstructed from PG
close the auction at exactly the scheduled time and reject bids that arrive after
auction close has a subtle correctness problem: a bid submitted at 23:59:59.999 and processed at 00:00:00.001 — is it valid? Weak answer: ignore bids after close time. Strong answer: explicit auction state machine: ACTIVE → CLOSING → CLOSED. On scheduled close time: set state to CLOSING (still accepts bids for a brief grace period). After grace period: set to CLOSED, reject all new bids. Determine winner from highest bid in PG
the auction close job must be idempotent — if it runs twice (due to retry), it should produce the same result. Implement with optimistic concurrency: UPDATE auctions SET status='CLOSED', winner_id=? WHERE id=? AND status='CLOSING'. If zero rows updated: another instance already closed it. Auction extension (eBay's approach): if a bid arrives in the last 3 minutes, extend the auction by 3 minutes. Implement: on any bid in the last 3 minutes, UPDATE auctions SET ends_at = ends_at + INTERVAL '3 minutes' WHERE id=? AND ends_at - NOW() < INTERVAL '3 minutes'
push every individual bid event to all watchers in real-time
an active auction with 10,000 watchers × 1 new bid every 10 seconds = 1,000 SSE messages/s for one auction. Naive fan-out: one Kafka message → 10,000 individual SSE pushes
delivery servers partitioned by auction_id (consistent hash). Each delivery server holds all SSE connections for its auction range. On new bid event: Kafka consumer on that delivery server pushes to all 10,000 connected clients in a tight loop. No cross-server coordination needed. For the last 30 seconds of a popular auction (bid storm): coalesce — buffer bids in 100ms windows, push the latest state (current highest bid) rather than every individual bid. Clients don't need every intermediate bid — they need the current highest bid to display correctly. This reduces fan-out from O(bids_per_second × watchers) to O(10 × watchers) during the final sprint
Why the deep dives connect to the scaling problem: "Hotspot writes, bid consistency, and real-time fan-out." Each deep dive addresses one dimension.
+-------------------+
| Clients |
| web mobile browser |
+---------+---------+
|
GET POST /auctions
POST /bids
SSE bid updates
|
+---------v---------+
| API Gateway |
| auth routing rate |
+----+----------+----+
| |
read path | | write path
| |
+--------------v--+ +--v----------------+
| Auction Service | | Bid Ingest API |
| auction details | | accept bid fast |
+-------+---------+ +---------+---------+
| |
| |
+--------v---------+ |
| Auctions DB |<---------------+
| auctions items | optional direct read
| max_bid on row |
+--------+---------+ |
^ |
| v
| +--------+---------+
| | Kafka |
| | topic partitioned|
| | by auctionId |
| +--------+---------+
| |
| v
| +--------+---------+
| | Bidding Service |
| | consumer workers |
| | validate bid |
| | OCC or row lock |
| +---+----------+---+
| | |
| | |
| write bid history |
| | |
| v v
| +------+--+ +---+----------------+
| | Bids DB | | Pub Sub / Fanout |
| | history | | broadcast updates |
| +---------+ +---+----------------+
| |
| |
+------v----------------------------------v------+
| SSE / Realtime Gateway instances |
| keep client connections by auctionId |
| push latest accepted max bid to watchers |
+----------------------+-------------------------+
|
v
+----+----+
| Clients |
| live UI |
+---------+The key idea is this. Reads and writes are split. Auction Service handles viewing and creating auctions. Bids go through a durable queue first so you do not lose them under spikes. Then Bidding Service processes bids in order per auction, updates the auction row max bid safely, stores bid history, and publishes the new highest bid to the realtime layer.
If you need to simplify this in an interview, keep the core path to four boxes. Client to API Gateway to Kafka to Bidding Service to Database, plus SSE for live updates. That shows you understand the two hard parts, which are correct bid acceptance and real-time fanout.
Problem
Accepting bids correctly at scale. Ensuring only valid higher bids win, no bids are lost, and everyone sees the current highest bid quickly.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is contention on a tiny piece of shared state. Many users may bid on the same auction at nearly the same time, but only one current highest bid can be correct.
That creates three scaling pain points. First, bid writes are hot and correctness matters, so you need atomic updates or version checks to avoid accepting stale lower bids. Second, auctions get bursty near the end, so one popular item can suddenly get hammered even if overall traffic looks manageable. Third, users expect live updates, which means one accepted bid may need to fan out quickly to many watchers across many servers. So the short interview answer is that Online Auction is hard because it combines hotspot writes, strict bid consistency, and real time fan-out.
Key points
- Scope it first. Core: list item, place bid, real-time bid feed, auction close, winner determination. Out of scope unless asked: payments, shipping, dispute resolution, seller ratings.
- Atomic bid check — Redis Lua CAS. GET current_max, compare, SET if higher — all in one Lua script. Single-threaded Redis execution means no two bids can race. Never use GET + SET as separate operations.
- Redis for speed, PostgreSQL for truth. Redis holds current max bid for sub-millisecond reads during live auction. PostgreSQL stores every bid for audit, dispute resolution, and winner determination. Both required.
- Auction close is a state machine. ACTIVE → CLOSING (stop accepting bids) → CLOSED (determine winner). Close job must be idempotent: UPDATE SET status=CLOSED WHERE status=CLOSING. If it runs twice, second is a no-op.
- Anti-sniping extension. Bid placed in last 3 minutes → extend auction by 3 minutes. UPDATE auctions SET ends_at = ends_at + INTERVAL 3 minutes WHERE id=? AND ends_at - NOW() < 3 minutes. Users prefer fair competition over fixed end time.
- Real-time bid feed — SSE with coalescing. 1K watchers × bid storm in last 30s = massive fan-out. 100ms coalescing: push current max bid state, not every individual bid. Watchers see the current price, not a log of every increment.
- Failure mode to name. Redis goes down mid-auction: freeze auction (reject new bids), reconstruct highest bid from PostgreSQL bid history, resume when Redis recovers. SLA: < 30s pause. PostgreSQL is always authoritative.
Tradeoffs
Deep dives
Two bidders submit simultaneously: A bids $100, B bids $102, both at the exact same millisecond. The system must accept $102 and reject $100
database transaction with SELECT FOR UPDATE
Redis Lua CAS (Compare-And-Set): (1) GET current_highest_bid, (2) if new_bid > current_highest, SET new_bid, return accepted, else return rejected. All atomic in one Lua script
Database transaction with SELECT FOR UPDATE is also correct but adds 5-20ms PG lock latency under high contention. Redis Lua CAS is <1ms. The tradeoff: Redis is not durable by default (RDB snapshot may be seconds old), so every accepted bid must also be written to PostgreSQL before returning success. The two-write pattern: Redis for speed and ordering, PG for durability. If Redis and PG diverge (Redis accepts a bid but PG write fails): on startup/recovery, the PG bid history is authoritative. Redis state is reconstructed from PG
close the auction at exactly the scheduled time and reject bids that arrive after
auction close has a subtle correctness problem: a bid submitted at 23:59:59.999 and processed at 00:00:00.001 — is it valid? Weak answer: ignore bids after close time. Strong answer: explicit auction state machine: ACTIVE → CLOSING → CLOSED. On scheduled close time: set state to CLOSING (still accepts bids for a brief grace period). After grace period: set to CLOSED, reject all new bids. Determine winner from highest bid in PG
the auction close job must be idempotent — if it runs twice (due to retry), it should produce the same result. Implement with optimistic concurrency: UPDATE auctions SET status='CLOSED', winner_id=? WHERE id=? AND status='CLOSING'. If zero rows updated: another instance already closed it. Auction extension (eBay's approach): if a bid arrives in the last 3 minutes, extend the auction by 3 minutes. Implement: on any bid in the last 3 minutes, UPDATE auctions SET ends_at = ends_at + INTERVAL '3 minutes' WHERE id=? AND ends_at - NOW() < INTERVAL '3 minutes'
push every individual bid event to all watchers in real-time
an active auction with 10,000 watchers × 1 new bid every 10 seconds = 1,000 SSE messages/s for one auction. Naive fan-out: one Kafka message → 10,000 individual SSE pushes
delivery servers partitioned by auction_id (consistent hash). Each delivery server holds all SSE connections for its auction range. On new bid event: Kafka consumer on that delivery server pushes to all 10,000 connected clients in a tight loop. No cross-server coordination needed. For the last 30 seconds of a popular auction (bid storm): coalesce — buffer bids in 100ms windows, push the latest state (current highest bid) rather than every individual bid. Clients don't need every intermediate bid — they need the current highest bid to display correctly. This reduces fan-out from O(bids_per_second × watchers) to O(10 × watchers) during the final sprint
Why the deep dives connect to the scaling problem: "Hotspot writes, bid consistency, and real-time fan-out." Each deep dive addresses one dimension.
Interview script
Whiteboard
+-------------------+
| Clients |
| web mobile browser |
+---------+---------+
|
GET POST /auctions
POST /bids
SSE bid updates
|
+---------v---------+
| API Gateway |
| auth routing rate |
+----+----------+----+
| |
read path | | write path
| |
+--------------v--+ +--v----------------+
| Auction Service | | Bid Ingest API |
| auction details | | accept bid fast |
+-------+---------+ +---------+---------+
| |
| |
+--------v---------+ |
| Auctions DB |<---------------+
| auctions items | optional direct read
| max_bid on row |
+--------+---------+ |
^ |
| v
| +--------+---------+
| | Kafka |
| | topic partitioned|
| | by auctionId |
| +--------+---------+
| |
| v
| +--------+---------+
| | Bidding Service |
| | consumer workers |
| | validate bid |
| | OCC or row lock |
| +---+----------+---+
| | |
| | |
| write bid history |
| | |
| v v
| +------+--+ +---+----------------+
| | Bids DB | | Pub Sub / Fanout |
| | history | | broadcast updates |
| +---------+ +---+----------------+
| |
| |
+------v----------------------------------v------+
| SSE / Realtime Gateway instances |
| keep client connections by auctionId |
| push latest accepted max bid to watchers |
+----------------------+-------------------------+
|
v
+----+----+
| Clients |
| live UI |
+---------+The key idea is this. Reads and writes are split. Auction Service handles viewing and creating auctions. Bids go through a durable queue first so you do not lose them under spikes. Then Bidding Service processes bids in order per auction, updates the auction row max bid safely, stores bid history, and publishes the new highest bid to the realtime layer.
If you need to simplify this in an interview, keep the core path to four boxes. Client to API Gateway to Kafka to Bidding Service to Database, plus SSE for live updates. That shows you understand the two hard parts, which are correct bid acceptance and real-time fanout.
assign products
per domain
dedup alerts
Helping users know whether an Amazon item is a good deal. Collect prices over time, show history charts, and alert when price drops below a threshold.
The hard part: getting accurate price data for a huge product catalog at scale.
The hard part is not storing prices. It is collecting and updating them at huge scale without overwhelming Amazon or your own system.
There are three main pain points. First, data collection is constrained. You may want to track 500 million products, but you cannot crawl them all frequently because Amazon rate limits scraping, so freshness becomes a resource allocation problem. Second, the workload is very uneven. A small set of products matters a lot more than the long tail, so you need prioritization based on user interest or extension traffic instead of treating every product the same. Third, one price change can create downstream fan-out. A single update may trigger validation, storage, chart updates, and notifications to many subscribers, so the system is not just a crawler. It is also an event processing pipeline.
A good interview summary is this. Price Tracking Service is hard to scale because data ingestion is externally constrained, freshness matters, and each accepted price update can fan out into a lot of follow-on work.
- Scope it first. Core: track product prices, store history, trigger alerts when price crosses threshold, display price charts. Out of scope unless asked: affiliate links, deal scoring, ML price prediction.
- Two collection strategies. Chrome extension: real-time, zero crawl cost, covers products users actively browse. Crawler: covers the long tail at lower frequency. Extension data for top 1M products, crawler for the rest.
- TSDB for price history — not SQL. Every price change is an immutable append. Queried by (product_id, time_range). InfluxDB or TimescaleDB: columnar, time-partitioned, compression. Plain PostgreSQL at 58 writes/sec × 500M products = wrong tool.
- Multi-resolution rollups. Raw price events: keep forever (small — only changes stored, not every poll). Daily rollup: min/max/avg per product per day. Chart rendering: raw for 7-day view, daily for 90-day, weekly for multi-year. Pre-computed by TimescaleDB continuous aggregates.
- Event-driven alerts. Price change → Kafka event → Alert Service evaluates watchlists for that product_id only. At 58 changes/sec × 5 alerts/product = 290 evaluations/sec. 1000× cheaper than polling all 100M alerts every 5 min.
- Trust but verify. Extension data can be wrong: A/B test pricing, regional variants, scraping errors. Fast-accept for display. Flag if new price differs >50% from previous. Priority-crawl flagged products for verification before storing permanently.
- Failure mode to name. Crawler gets IP-banned: rotate residential proxies, respect Crawl-delay, randomize timing. Extension-first strategy means the most important products still get real-time data even if crawler is blocked.
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
crawl Amazon every hour for all tracked products
two-tier collection — Chrome extension gives real-time prices for user-active products at zero crawl cost; crawler covers the long tail at lower frequency. The extension is the primary path for the top 1M products users actually care about
extension data can be wrong — A/B test prices, regional variants, scraping bugs. Fast-accept for display, but flag price changes >50% from previous as anomalies and trigger a priority crawler re-fetch before storing permanently. This two-layer validation gives low latency for normal updates and high confidence for outliers
cron job every 5 minutes queries all 100M active alerts for products that changed price. This is O(active_alerts) per run = 333K DB queries per second just for alerts. Unacceptable
event-driven evaluation. When a price update is stored, publish a Kafka event {product_id, old_price, new_price}. Alert service consumes the event and evaluates only alerts for that product_id. At 58 changes/sec × avg 5 alerts/product = 290 evaluations/sec — 1000× more efficient
the dedup contract — don't re-alert if the price hasn't crossed the threshold fresh. Track last_alert_sent_at per alert; require the price to recover above threshold and drop below again before re-alerting. Without this, a price hovering at the threshold triggers a new alert on every crawl cycle
store every price event in PostgreSQL, query by product_id and time range
InfluxDB or TimescaleDB — append-only, time-partitioned, columnar compression. But the real win is multi-resolution rollups: raw price events for all-time history, daily min/max/avg for chart display. 7-day view uses raw data; 90-day uses daily rollup; 3-year uses weekly rollup
pre-compute rollups with TimescaleDB continuous aggregates — they update automatically on every insert, so a 3-year chart query fetches 156 weekly rows instead of 26K raw events. 170× faster query with no visible chart difference at typical chart widths
Why the deep dives connect to the scaling problem: "Data collection at scale, event-driven processing, and efficient time-series queries." Each deep dive solves one challenge.
+----------------------+
| Website / Chrome |
| Extension |
+----------+-----------+
|
v
+------------------+
| API Gateway |
| auth rate limit |
+---+----------+---+
| |
GET price hist | | POST subscription
v v
+----------------+ +-------------------+
| Price History | | Subscription |
| Service | | Service |
+-------+--------+ +---------+---------+
| |
v v
+---------------------+ +---------------------+
| Price DB | | Primary DB |
| time series prices | | users products subs |
+----------+----------+ +----------+----------+
| |
| new validated price |
v |
+-------------------+ |
| Kafka / Event Bus |<----------+
| price change evt |
+---------+---------+
|
v
+--------------------------+
| Notification Service |
| find matching subs |
+------------+-------------+
|
v
+----------------------+
| Email Provider |
+----------------------+
PRICE COLLECTION SIDE
+----------------------+ +----------------------+
| Chrome Extension | | Web Crawler Service |
| product page views | | selective crawling |
+----------+-----------+ +----------+-----------+
| |
v v
+-----------------------------------------------+
| Price Ingestion / Validation Service |
| trust but verify suspicious updates |
+-------------------+---------------------------+
|
valid price | write
v
+--------------+
| Price DB |
+------+--------+
|
| publish price changed
v
+--------------+
| Kafka / Bus |
+--------------+
OPTIONAL FAST VERIFICATION LOOP
suspicious extension update
|
v
+------------------------------+
| Verification Queue |
+--------------+---------------+
|
v
+------------------------------+
| Priority Crawler |
| checks Amazon quickly |
+--------------+---------------+
|
v
+------------------------------+
| Validation Service updates |
| trust score and final price |
+------------------------------+The main idea is this. Extension plus crawler collect prices, validation decides what to trust, validated price changes go into the price database, and those changes produce events that drive notifications. Separately, the read path for charts stays simple and fast through the Price History Service querying the time series price store.
If you are drawing this in an interview, I would start with just three lanes. Client API read path, data collection path, and notification path. That keeps the whiteboard clean and makes the story easy to explain.
Problem
Helping users know whether an Amazon item is a good deal. Collect prices over time, show history charts, and alert when price drops below a threshold.
The hard part: getting accurate price data for a huge product catalog at scale.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is not storing prices. It is collecting and updating them at huge scale without overwhelming Amazon or your own system.
There are three main pain points. First, data collection is constrained. You may want to track 500 million products, but you cannot crawl them all frequently because Amazon rate limits scraping, so freshness becomes a resource allocation problem. Second, the workload is very uneven. A small set of products matters a lot more than the long tail, so you need prioritization based on user interest or extension traffic instead of treating every product the same. Third, one price change can create downstream fan-out. A single update may trigger validation, storage, chart updates, and notifications to many subscribers, so the system is not just a crawler. It is also an event processing pipeline.
A good interview summary is this. Price Tracking Service is hard to scale because data ingestion is externally constrained, freshness matters, and each accepted price update can fan out into a lot of follow-on work.
Key points
- Scope it first. Core: track product prices, store history, trigger alerts when price crosses threshold, display price charts. Out of scope unless asked: affiliate links, deal scoring, ML price prediction.
- Two collection strategies. Chrome extension: real-time, zero crawl cost, covers products users actively browse. Crawler: covers the long tail at lower frequency. Extension data for top 1M products, crawler for the rest.
- TSDB for price history — not SQL. Every price change is an immutable append. Queried by (product_id, time_range). InfluxDB or TimescaleDB: columnar, time-partitioned, compression. Plain PostgreSQL at 58 writes/sec × 500M products = wrong tool.
- Multi-resolution rollups. Raw price events: keep forever (small — only changes stored, not every poll). Daily rollup: min/max/avg per product per day. Chart rendering: raw for 7-day view, daily for 90-day, weekly for multi-year. Pre-computed by TimescaleDB continuous aggregates.
- Event-driven alerts. Price change → Kafka event → Alert Service evaluates watchlists for that product_id only. At 58 changes/sec × 5 alerts/product = 290 evaluations/sec. 1000× cheaper than polling all 100M alerts every 5 min.
- Trust but verify. Extension data can be wrong: A/B test pricing, regional variants, scraping errors. Fast-accept for display. Flag if new price differs >50% from previous. Priority-crawl flagged products for verification before storing permanently.
- Failure mode to name. Crawler gets IP-banned: rotate residential proxies, respect Crawl-delay, randomize timing. Extension-first strategy means the most important products still get real-time data even if crawler is blocked.
Tradeoffs
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
crawl Amazon every hour for all tracked products
two-tier collection — Chrome extension gives real-time prices for user-active products at zero crawl cost; crawler covers the long tail at lower frequency. The extension is the primary path for the top 1M products users actually care about
extension data can be wrong — A/B test prices, regional variants, scraping bugs. Fast-accept for display, but flag price changes >50% from previous as anomalies and trigger a priority crawler re-fetch before storing permanently. This two-layer validation gives low latency for normal updates and high confidence for outliers
cron job every 5 minutes queries all 100M active alerts for products that changed price. This is O(active_alerts) per run = 333K DB queries per second just for alerts. Unacceptable
event-driven evaluation. When a price update is stored, publish a Kafka event {product_id, old_price, new_price}. Alert service consumes the event and evaluates only alerts for that product_id. At 58 changes/sec × avg 5 alerts/product = 290 evaluations/sec — 1000× more efficient
the dedup contract — don't re-alert if the price hasn't crossed the threshold fresh. Track last_alert_sent_at per alert; require the price to recover above threshold and drop below again before re-alerting. Without this, a price hovering at the threshold triggers a new alert on every crawl cycle
store every price event in PostgreSQL, query by product_id and time range
InfluxDB or TimescaleDB — append-only, time-partitioned, columnar compression. But the real win is multi-resolution rollups: raw price events for all-time history, daily min/max/avg for chart display. 7-day view uses raw data; 90-day uses daily rollup; 3-year uses weekly rollup
pre-compute rollups with TimescaleDB continuous aggregates — they update automatically on every insert, so a 3-year chart query fetches 156 weekly rows instead of 26K raw events. 170× faster query with no visible chart difference at typical chart widths
Why the deep dives connect to the scaling problem: "Data collection at scale, event-driven processing, and efficient time-series queries." Each deep dive solves one challenge.
Interview script
Whiteboard
+----------------------+
| Website / Chrome |
| Extension |
+----------+-----------+
|
v
+------------------+
| API Gateway |
| auth rate limit |
+---+----------+---+
| |
GET price hist | | POST subscription
v v
+----------------+ +-------------------+
| Price History | | Subscription |
| Service | | Service |
+-------+--------+ +---------+---------+
| |
v v
+---------------------+ +---------------------+
| Price DB | | Primary DB |
| time series prices | | users products subs |
+----------+----------+ +----------+----------+
| |
| new validated price |
v |
+-------------------+ |
| Kafka / Event Bus |<----------+
| price change evt |
+---------+---------+
|
v
+--------------------------+
| Notification Service |
| find matching subs |
+------------+-------------+
|
v
+----------------------+
| Email Provider |
+----------------------+
PRICE COLLECTION SIDE
+----------------------+ +----------------------+
| Chrome Extension | | Web Crawler Service |
| product page views | | selective crawling |
+----------+-----------+ +----------+-----------+
| |
v v
+-----------------------------------------------+
| Price Ingestion / Validation Service |
| trust but verify suspicious updates |
+-------------------+---------------------------+
|
valid price | write
v
+--------------+
| Price DB |
+------+--------+
|
| publish price changed
v
+--------------+
| Kafka / Bus |
+--------------+
OPTIONAL FAST VERIFICATION LOOP
suspicious extension update
|
v
+------------------------------+
| Verification Queue |
+--------------+---------------+
|
v
+------------------------------+
| Priority Crawler |
| checks Amazon quickly |
+--------------+---------------+
|
v
+------------------------------+
| Validation Service updates |
| trust score and final price |
+------------------------------+The main idea is this. Extension plus crawler collect prices, validation decides what to trust, validated price changes go into the price database, and those changes produce events that drive notifications. Separately, the read path for charts stays simple and fast through the Price History Service querying the time series price store.
If you are drawing this in an interview, I would start with just three lanes. Client API read path, data collection path, and notification path. That keeps the whiteboard clean and makes the story easy to explain.
· alerts
rate limit
SMS · Email
Send millions of notifications daily across push, SMS, and email with delivery guarantees, user preference respect, and survival of viral fan-out spikes.
Hard parts: multi-channel routing, at-least-once deduplication, and third-party provider rate limits during events.
The hard part is fan-out: one viral event creates millions of deliveries, but third-party providers cap throughput. Kafka handles ingestion; workers and staggered delivery handle the bottleneck.
- Scope. Core: send push/SMS/email, respect opt-outs, retry failures, track delivery status. Out of scope unless asked: in-app inbox, rich media templates, A/B testing.
- Async by default. API enqueues and returns immediately. Never block the caller on APNs/FCM latency.
- Per-channel Kafka topics. ios-push, android-push, sms, email — independent worker pools and retry policies.
- Dedup on event_id. SETNX event_id with 24h TTL before send. Prevents duplicate notifications on Kafka replay.
- Preferences before queue. Cache opt-outs in Redis. Filter before publishing — never waste queue capacity on opted-out users.
- Two-tier priority. Transactional (password reset, purchase) bypasses marketing rate limits. Marketing respects max N/user/hour.
- Invalid token cleanup. On APNs BadDeviceToken: delete token immediately. Never retry dead endpoints.
one worker pool handles all channels
channel-specific workers because rate limits, payload formats, and failure modes differ. APNs uses device tokens and certificate auth; email uses SMTP/API with bounce handling; SMS has per-country regulations
partition Kafka by user_id hash for parallelism while keeping per-user ordering within a channel. Critical notifications use a dedicated high-priority topic with reserved worker capacity
Worker crashes after sending but before committing offset → message redelivered → duplicate notification. Redis SETNX on event_id before send. If key exists, skip. TTL = 24h covers replay window
Retry until delivery succeeds — duplicates are rare.
Worker crashes after sending but before committing offset → message redelivered → duplicate notification. Redis SETNX on event_id before send. If key exists, skip. TTL = 24h covers replay window
include idempotency key in provider request (FCM collapse_key) so the provider also deduplicates. Document contract: delivery is at-least-once; consumers must be idempotent
One event → 10M notifications in 60s. Kafka absorbs the write spike, but APNs/FCM rate-limit per certificate. Stagger enqueue over 60–120s. Monitor provider 429 responses and backoff globally. Batch similar notifications (5 new likes → one grouped push)
Push to every device synchronously from the API handler.
One event → 10M notifications in 60s. Kafka absorbs the write spike, but APNs/FCM rate-limit per certificate. Stagger enqueue over 60–120s. Monitor provider 429 responses and backoff globally. Batch similar notifications (5 new likes → one grouped push)
APNs coalesces offline notifications — only latest is delivered. For badge counts, send silent data push that triggers app to fetch true count from API
Delete device tokens immediately. Publish user_deleted event. All workers check Redis blocklist before send. Kafka messages for deleted users cannot be erased — skip at dispatch. Document 72h purge SLA for compliance
Delete the user row — async workers will stop eventually.
Delete device tokens immediately. Publish user_deleted event. All workers check Redis blocklist before send. Kafka messages for deleted users cannot be erased — skip at dispatch. Document 72h purge SLA for compliance
Name metric + revisit trigger when they push depth.
Three-problem script.
+------------------+
| Trigger sources |
| txn · marketing |
+--------+---------+
|
v
+------------------+
| Notification API |
| prefs · dedup |
+--------+---------+
|
+--------------+--------------+
| | |
v v v
+-----------+ +-----------+ +-----------+
| Kafka iOS | |Kafka SMS | |Kafka Email|
+-----+-----+ +-----+-----+ +-----+-----+
| | |
v v v
+-----------+ +-----------+ +-----------+
| iOS worker| | SMS worker| |Email worker|
+-----+-----+ +-----+-----+ +-----+-----+
| | |
v v v
APNs/FCM Twilio SendGridInterview version: API checks prefs and dedup, publishes to Kafka, workers call providers. Add DLQ and token cleanup if pushed on reliability.
Problem
Send millions of notifications daily across push, SMS, and email with delivery guarantees, user preference respect, and survival of viral fan-out spikes.
Hard parts: multi-channel routing, at-least-once deduplication, and third-party provider rate limits during events.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is fan-out: one viral event creates millions of deliveries, but third-party providers cap throughput. Kafka handles ingestion; workers and staggered delivery handle the bottleneck.
Key points
- Scope. Core: send push/SMS/email, respect opt-outs, retry failures, track delivery status. Out of scope unless asked: in-app inbox, rich media templates, A/B testing.
- Async by default. API enqueues and returns immediately. Never block the caller on APNs/FCM latency.
- Per-channel Kafka topics. ios-push, android-push, sms, email — independent worker pools and retry policies.
- Dedup on event_id. SETNX event_id with 24h TTL before send. Prevents duplicate notifications on Kafka replay.
- Preferences before queue. Cache opt-outs in Redis. Filter before publishing — never waste queue capacity on opted-out users.
- Two-tier priority. Transactional (password reset, purchase) bypasses marketing rate limits. Marketing respects max N/user/hour.
- Invalid token cleanup. On APNs BadDeviceToken: delete token immediately. Never retry dead endpoints.
Tradeoffs
Deep dives
one worker pool handles all channels
channel-specific workers because rate limits, payload formats, and failure modes differ. APNs uses device tokens and certificate auth; email uses SMTP/API with bounce handling; SMS has per-country regulations
partition Kafka by user_id hash for parallelism while keeping per-user ordering within a channel. Critical notifications use a dedicated high-priority topic with reserved worker capacity
Worker crashes after sending but before committing offset → message redelivered → duplicate notification. Redis SETNX on event_id before send. If key exists, skip. TTL = 24h covers replay window
Retry until delivery succeeds — duplicates are rare.
Worker crashes after sending but before committing offset → message redelivered → duplicate notification. Redis SETNX on event_id before send. If key exists, skip. TTL = 24h covers replay window
include idempotency key in provider request (FCM collapse_key) so the provider also deduplicates. Document contract: delivery is at-least-once; consumers must be idempotent
One event → 10M notifications in 60s. Kafka absorbs the write spike, but APNs/FCM rate-limit per certificate. Stagger enqueue over 60–120s. Monitor provider 429 responses and backoff globally. Batch similar notifications (5 new likes → one grouped push)
Push to every device synchronously from the API handler.
One event → 10M notifications in 60s. Kafka absorbs the write spike, but APNs/FCM rate-limit per certificate. Stagger enqueue over 60–120s. Monitor provider 429 responses and backoff globally. Batch similar notifications (5 new likes → one grouped push)
APNs coalesces offline notifications — only latest is delivered. For badge counts, send silent data push that triggers app to fetch true count from API
Delete device tokens immediately. Publish user_deleted event. All workers check Redis blocklist before send. Kafka messages for deleted users cannot be erased — skip at dispatch. Document 72h purge SLA for compliance
Delete the user row — async workers will stop eventually.
Delete device tokens immediately. Publish user_deleted event. All workers check Redis blocklist before send. Kafka messages for deleted users cannot be erased — skip at dispatch. Document 72h purge SLA for compliance
Name metric + revisit trigger when they push depth.
Interview script
Three-problem script.
Whiteboard
+------------------+
| Trigger sources |
| txn · marketing |
+--------+---------+
|
v
+------------------+
| Notification API |
| prefs · dedup |
+--------+---------+
|
+--------------+--------------+
| | |
v v v
+-----------+ +-----------+ +-----------+
| Kafka iOS | |Kafka SMS | |Kafka Email|
+-----+-----+ +-----+-----+ +-----+-----+
| | |
v v v
+-----------+ +-----------+ +-----------+
| iOS worker| | SMS worker| |Email worker|
+-----+-----+ +-----+-----+ +-----+-----+
| | |
v v v
APNs/FCM Twilio SendGridInterview version: API checks prefs and dedup, publishes to Kafka, workers call providers. Add DLQ and token cleanup if pushed on reliability.
Return top-10 search completions within 100ms as the user types, at billions of queries per day.
Hard parts: sub-100ms latency, fresh trending terms, and trie size small enough to serve from cache.
Autocomplete looks read-heavy but the real trick is avoiding work: debounce cuts QPS 10×, CDN absorbs 90%, trie pre-computation eliminates subtree scans.
- Trie with top-K per node. Traverse to prefix node, return cached top-10 — no subtree walk.
- Batch rebuild. Do not update trie on every query. Weekly full rebuild + hot-term injection for breaking news.
- Redis serving. Key = prefix, value = suggestions array. Sub-ms lookup.
- CDN for hot prefixes. Top 10K prefixes cover ~90% of traffic.
- Client debounce. 100–200ms debounce cuts backend QPS ~10×.
- Shard by prefix. First character (or 2-gram) → independent Redis shard.
- Content filter. Blocklist inappropriate suggestions before storing in trie.
scan all queries matching prefix on every keystroke
prefix tree where each node stores top-10 completions by global frequency. Query = O(prefix length) traversal + O(1) return
1M unique prefixes × 10 suggestions × 50B ≈ 500 MB in Redis
Weekly MapReduce over query logs → frequency table → trie builder → shadow deploy → atomic flip. Breaking news: real-time detector flags queries with no trie match exceeding 1K/5min → inject temporary hot entry until next rebuild
Rebuild the full index nightly — no incremental updates.
Weekly MapReduce over query logs → frequency table → trie builder → shadow deploy → atomic flip. Breaking news: real-time detector flags queries with no trie match exceeding 1K/5min → inject temporary hot entry until next rebuild
Name metric + revisit trigger when they push depth.
578K QPS raw keystrokes → 58K with debounce. CDN serves 90% from edge. Backend sees ~5.8K QPS for long tail. Shard Redis by first character; sub-shard hot prefixes like "th"
Serve every request from origin — CDN is optional.
578K QPS raw keystrokes → 58K with debounce. CDN serves 90% from edge. Backend sees ~5.8K QPS for long tail. Shard Redis by first character; sub-shard hot prefixes like "th"
Name metric + revisit trigger when they push depth.
Fetch user boost vector from Redis (recent searches). Re-rank trie top-50 candidates in ~5ms. Decouple retrieval (trie) from ranking (lightweight model)
Build a per-user trie — one per user at scale.
Fetch user boost vector from Redis (recent searches). Re-rank trie top-50 candidates in ~5ms. Decouple retrieval (trie) from ranking (lightweight model)
Name metric + revisit trigger when they push depth.
[Query logs] --> [MapReduce] --> [Trie Builder] --> [Redis shards]
^
[User keystroke] --> [Debounce] --> [API] --+--> [CDN hit?] --> return
| miss
v
[Redis prefix lookup]Say offline build + online lookup. CDN and debounce are the scaling story.
Problem
Return top-10 search completions within 100ms as the user types, at billions of queries per day.
Hard parts: sub-100ms latency, fresh trending terms, and trie size small enough to serve from cache.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
Autocomplete looks read-heavy but the real trick is avoiding work: debounce cuts QPS 10×, CDN absorbs 90%, trie pre-computation eliminates subtree scans.
Key points
- Trie with top-K per node. Traverse to prefix node, return cached top-10 — no subtree walk.
- Batch rebuild. Do not update trie on every query. Weekly full rebuild + hot-term injection for breaking news.
- Redis serving. Key = prefix, value = suggestions array. Sub-ms lookup.
- CDN for hot prefixes. Top 10K prefixes cover ~90% of traffic.
- Client debounce. 100–200ms debounce cuts backend QPS ~10×.
- Shard by prefix. First character (or 2-gram) → independent Redis shard.
- Content filter. Blocklist inappropriate suggestions before storing in trie.
Tradeoffs
Deep dives
scan all queries matching prefix on every keystroke
prefix tree where each node stores top-10 completions by global frequency. Query = O(prefix length) traversal + O(1) return
1M unique prefixes × 10 suggestions × 50B ≈ 500 MB in Redis
Weekly MapReduce over query logs → frequency table → trie builder → shadow deploy → atomic flip. Breaking news: real-time detector flags queries with no trie match exceeding 1K/5min → inject temporary hot entry until next rebuild
Rebuild the full index nightly — no incremental updates.
Weekly MapReduce over query logs → frequency table → trie builder → shadow deploy → atomic flip. Breaking news: real-time detector flags queries with no trie match exceeding 1K/5min → inject temporary hot entry until next rebuild
Name metric + revisit trigger when they push depth.
578K QPS raw keystrokes → 58K with debounce. CDN serves 90% from edge. Backend sees ~5.8K QPS for long tail. Shard Redis by first character; sub-shard hot prefixes like "th"
Serve every request from origin — CDN is optional.
578K QPS raw keystrokes → 58K with debounce. CDN serves 90% from edge. Backend sees ~5.8K QPS for long tail. Shard Redis by first character; sub-shard hot prefixes like "th"
Name metric + revisit trigger when they push depth.
Fetch user boost vector from Redis (recent searches). Re-rank trie top-50 candidates in ~5ms. Decouple retrieval (trie) from ranking (lightweight model)
Build a per-user trie — one per user at scale.
Fetch user boost vector from Redis (recent searches). Re-rank trie top-50 candidates in ~5ms. Decouple retrieval (trie) from ranking (lightweight model)
Name metric + revisit trigger when they push depth.
Interview script
Whiteboard
[Query logs] --> [MapReduce] --> [Trie Builder] --> [Redis shards]
^
[User keystroke] --> [Debounce] --> [API] --+--> [CDN hit?] --> return
| miss
v
[Redis prefix lookup]Say offline build + online lookup. CDN and debounce are the scaling story.
Generate unique, roughly time-ordered IDs at high throughput across thousands of servers.
Hard parts: coordination without per-ID DB calls, clock drift, and sequence overflow within one millisecond.
ID generation is embarrassingly parallel once worker_ids are assigned. The coordination pain is clock sync and worker_id leasing, not throughput.
- 64-bit layout. 41b timestamp | 10b worker_id | 12b sequence = 4096 IDs/ms per worker.
- Worker ID lease. ZooKeeper/etcd assigns worker_id. Ephemeral node — reclaimed on crash.
- Clock synchronization. NTP required. If clock moves backward: wait until caught up or fail loudly — never emit duplicate IDs.
- Sequence per millisecond. INCR sequence within same ms. Rollover → wait next ms.
- No DB per ID. Generation is in-process after worker_id assigned — millions/sec per node.
- Sortable. Time-ordered IDs useful for sharding and debugging.
- Not cryptographic. IDs are predictable — do not use as security tokens.
12-bit sequence = 4096 IDs/ms per worker. 10-bit worker = 1024 machines. Cluster theoretical max ≈ 4M IDs/ms
use UUID
explain bit allocation and why sortability helps DB indexing
Name the metric you'd alert on and when you'd revisit this design.
NTP sync required. If current_ms < last_ms: wait or error. Never reuse timestamp+sequence combo
Use system clock on each machine — NTP is optional.
NTP sync required. If current_ms < last_ms: wait or error. Never reuse timestamp+sequence combo
leap seconds and VM migration can move clock backward — monitor and alert
ZooKeeper ephemeral sequential nodes assign worker_id. On crash, ID reclaimed after session timeout. Alternative: DB lease table with heartbeat — slower failover
Pick a random worker ID at process start.
ZooKeeper ephemeral sequential nodes assign worker_id. On crash, ID reclaimed after session timeout. Alternative: DB lease table with heartbeat — slower failover
Name metric + revisit trigger when they push depth.
Per-DC worker_id ranges avoid cross-DC ZK dependency. Or dedicated ID service per region with DC bits in layout
UUID v4 everywhere — collisions are negligible.
Per-DC worker_id ranges avoid cross-DC ZK dependency. Or dedicated ID service per region with DC bits in layout
Name metric + revisit trigger when they push depth.
[Service] --> [ID Generator lib]
|
worker_id from ZK lease
seq++ per millisecond
pack 64-bit IDDraw three fields in the 64-bit ID. ZK assigns worker_id. Generation is local after lease.
Problem
Generate unique, roughly time-ordered IDs at high throughput across thousands of servers.
Hard parts: coordination without per-ID DB calls, clock drift, and sequence overflow within one millisecond.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
ID generation is embarrassingly parallel once worker_ids are assigned. The coordination pain is clock sync and worker_id leasing, not throughput.
Key points
- 64-bit layout. 41b timestamp | 10b worker_id | 12b sequence = 4096 IDs/ms per worker.
- Worker ID lease. ZooKeeper/etcd assigns worker_id. Ephemeral node — reclaimed on crash.
- Clock synchronization. NTP required. If clock moves backward: wait until caught up or fail loudly — never emit duplicate IDs.
- Sequence per millisecond. INCR sequence within same ms. Rollover → wait next ms.
- No DB per ID. Generation is in-process after worker_id assigned — millions/sec per node.
- Sortable. Time-ordered IDs useful for sharding and debugging.
- Not cryptographic. IDs are predictable — do not use as security tokens.
Tradeoffs
Deep dives
12-bit sequence = 4096 IDs/ms per worker. 10-bit worker = 1024 machines. Cluster theoretical max ≈ 4M IDs/ms
use UUID
explain bit allocation and why sortability helps DB indexing
Name the metric you'd alert on and when you'd revisit this design.
NTP sync required. If current_ms < last_ms: wait or error. Never reuse timestamp+sequence combo
Use system clock on each machine — NTP is optional.
NTP sync required. If current_ms < last_ms: wait or error. Never reuse timestamp+sequence combo
leap seconds and VM migration can move clock backward — monitor and alert
ZooKeeper ephemeral sequential nodes assign worker_id. On crash, ID reclaimed after session timeout. Alternative: DB lease table with heartbeat — slower failover
Pick a random worker ID at process start.
ZooKeeper ephemeral sequential nodes assign worker_id. On crash, ID reclaimed after session timeout. Alternative: DB lease table with heartbeat — slower failover
Name metric + revisit trigger when they push depth.
Per-DC worker_id ranges avoid cross-DC ZK dependency. Or dedicated ID service per region with DC bits in layout
UUID v4 everywhere — collisions are negligible.
Per-DC worker_id ranges avoid cross-DC ZK dependency. Or dedicated ID service per region with DC bits in layout
Name metric + revisit trigger when they push depth.
Interview script
Whiteboard
[Service] --> [ID Generator lib]
|
worker_id from ZK lease
seq++ per millisecond
pack 64-bit IDDraw three fields in the 64-bit ID. ZK assigns worker_id. Generation is local after lease.
Search hotels, book rooms, prevent double-booking when two users grab the last room.
Correctness dominates — 167 bookings/sec is easy; double-book is catastrophic.
Hotel reservation is a correctness problem. Contention spikes on popular dates; search volume dwarfs bookings but must not touch OLTP locks.
- Inventory model. room_inventory(hotel, type, date, total, reserved). Available = total - reserved.
- Pessimistic lock. SELECT FOR UPDATE on inventory row during booking transaction.
- Idempotency key. Client UUID — retry returns same reservation, no double charge.
- Search vs book separation. Search is approximate/fast (ES + Redis). Booking hits authoritative MySQL.
- 10-minute hold. PENDING_PAYMENT reservation occupies slot. Background job expires holds.
- Saga. Reserve sync → pay async → confirm. Compensate (release inventory) if pay fails.
- Cache advisory only. Stale cache may show availability; DB lock prevents double-book.
Transaction: SELECT reserved FROM inventory WHERE hotel=X AND date=Y FOR UPDATE. Check available>0. UPDATE reserved++. INSERT reservation. COMMIT. Second transaction blocks until first completes — sees updated count
UPDATE balance in SQL — no locking story.
Transaction: SELECT reserved FROM inventory WHERE hotel=X AND date=Y FOR UPDATE. Check available>0. UPDATE reserved++. INSERT reservation. COMMIT. Second transaction blocks until first completes — sees updated count
Name metric + revisit trigger when they push depth.
Lock held only for transaction duration (~50ms). PENDING_PAYMENT row holds inventory. Expiry job releases after 10 min. Payment at T+9:59 still valid if timestamp authoritative
Retry the charge on any timeout.
Lock held only for transaction duration (~50ms). PENDING_PAYMENT row holds inventory. Expiry job releases after 10 min. Payment at T+9:59 still valid if timestamp authoritative
Name metric + revisit trigger when they push depth.
Elasticsearch for hotel metadata/ranking. Redis for availability counts updated on booking. Search never acquires row locks
SELECT * WHERE column LIKE '%query%'.
Elasticsearch for hotel metadata/ranking. Redis for availability counts updated on booking. Search never acquires row locks
Name metric + revisit trigger when they push depth.
Pay fails → cancel reservation → decrement reserved. Idempotent compensation keyed by reservation_id
Oversimplify saga compensation — name one component, skip failure modes and metrics.
Pay fails → cancel reservation → decrement reserved. Idempotent compensation keyed by reservation_id
Name metric + revisit trigger when they push depth.
Search: User -> ES/Redis (approximate) Book: User -> Booking Svc -> PG FOR UPDATE -> Kafka -> Payment -> Confirm
Two paths: fast approximate search, exact transactional booking.
Problem
Search hotels, book rooms, prevent double-booking when two users grab the last room.
Correctness dominates — 167 bookings/sec is easy; double-book is catastrophic.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
Hotel reservation is a correctness problem. Contention spikes on popular dates; search volume dwarfs bookings but must not touch OLTP locks.
Key points
- Inventory model. room_inventory(hotel, type, date, total, reserved). Available = total - reserved.
- Pessimistic lock. SELECT FOR UPDATE on inventory row during booking transaction.
- Idempotency key. Client UUID — retry returns same reservation, no double charge.
- Search vs book separation. Search is approximate/fast (ES + Redis). Booking hits authoritative MySQL.
- 10-minute hold. PENDING_PAYMENT reservation occupies slot. Background job expires holds.
- Saga. Reserve sync → pay async → confirm. Compensate (release inventory) if pay fails.
- Cache advisory only. Stale cache may show availability; DB lock prevents double-book.
Tradeoffs
Deep dives
Transaction: SELECT reserved FROM inventory WHERE hotel=X AND date=Y FOR UPDATE. Check available>0. UPDATE reserved++. INSERT reservation. COMMIT. Second transaction blocks until first completes — sees updated count
UPDATE balance in SQL — no locking story.
Transaction: SELECT reserved FROM inventory WHERE hotel=X AND date=Y FOR UPDATE. Check available>0. UPDATE reserved++. INSERT reservation. COMMIT. Second transaction blocks until first completes — sees updated count
Name metric + revisit trigger when they push depth.
Lock held only for transaction duration (~50ms). PENDING_PAYMENT row holds inventory. Expiry job releases after 10 min. Payment at T+9:59 still valid if timestamp authoritative
Retry the charge on any timeout.
Lock held only for transaction duration (~50ms). PENDING_PAYMENT row holds inventory. Expiry job releases after 10 min. Payment at T+9:59 still valid if timestamp authoritative
Name metric + revisit trigger when they push depth.
Elasticsearch for hotel metadata/ranking. Redis for availability counts updated on booking. Search never acquires row locks
SELECT * WHERE column LIKE '%query%'.
Elasticsearch for hotel metadata/ranking. Redis for availability counts updated on booking. Search never acquires row locks
Name metric + revisit trigger when they push depth.
Pay fails → cancel reservation → decrement reserved. Idempotent compensation keyed by reservation_id
Oversimplify saga compensation — name one component, skip failure modes and metrics.
Pay fails → cancel reservation → decrement reserved. Idempotent compensation keyed by reservation_id
Name metric + revisit trigger when they push depth.
Interview script
Whiteboard
Search: User -> ES/Redis (approximate) Book: User -> Booking Svc -> PG FOR UPDATE -> Kafka -> Payment -> Confirm
Two paths: fast approximate search, exact transactional booking.
Real-time global leaderboard for millions of players: update scores instantly, return top-100 and any player rank fast.
Redis sorted set is purpose-built — DB RANK() scans are too slow at 100M players.
Rank queries stay O(log N) as N grows. Real limits are write throughput during tournaments and top-100 read storms — both solved with ZINCRBY and short TTL cache.
- Redis sorted set core. ZADD/ZINCRBY/ZREVRANK/ZREVRANGE — all O(log N).
- ZINCRBY not read-modify-write. Atomic increment — no race on concurrent match results.
- Cassandra durability. Redis is serving layer; rebuild ZSET from Cassandra on failure.
- Top-100 cache. Cache ZREVRANGE result 1s TTL — tournament end storm.
- Multiple boards. Global, regional, friends, seasonal — separate ZSETs updated in pipeline.
- Rank around me. ZREVRANK then ZREVRANGE R-5 R+5.
- Server-side validation. Anti-cheat: validate score server-side before ZINCRBY.
At 100M players RANK() OVER scans entire table. ZREVRANK is ~27 comparisons. Top-100 is O(100) regardless of N
Oversimplify why sorted set beats sql — name one component, skip failure modes and metrics.
At 100M players RANK() OVER scans entire table. ZREVRANK is ~27 comparisons. Top-100 is O(100) regardless of N
Name metric + revisit trigger when they push depth.
Redis AOF + RDB. On total loss: batch ZADD from Cassandra — minutes for 100M players. Serve stale cached top-100 during rebuild
Oversimplify durability and rebuild — name one component, skip failure modes and metrics.
Redis AOF + RDB. On total loss: batch ZADD from Cassandra — minutes for 100M players. Serve stale cached top-100 during rebuild
Name metric + revisit trigger when they push depth.
167K ZINCRBY/s within Redis capacity. Top-100 cache with 1s TTL collapses read storm to 1 ZREVRANGE/s
Oversimplify write throughput and tournament storms — name one component, skip failure modes and metrics.
167K ZINCRBY/s within Redis capacity. Top-100 cache with 1s TTL collapses read storm to 1 ZREVRANGE/s
Name metric + revisit trigger when they push depth.
RENAME leaderboard:daily to leaderboard:daily:yesterday atomically at midnight. New empty set for new period
Oversimplify seasonal resets without downtime — name one component, skip failure modes and metrics.
RENAME leaderboard:daily to leaderboard:daily:yesterday atomically at midnight. New empty set for new period
Name metric + revisit trigger when they push depth.
Match -> Score Service -> ZINCRBY leaderboard
|-> Cassandra (audit)
API -> ZREVRANGE top-100 (cached)
API -> ZREVRANK player_idOne diagram: write path ZINCRBY, read paths ZREVRANGE and ZREVRANK.
Problem
Real-time global leaderboard for millions of players: update scores instantly, return top-100 and any player rank fast.
Redis sorted set is purpose-built — DB RANK() scans are too slow at 100M players.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
Rank queries stay O(log N) as N grows. Real limits are write throughput during tournaments and top-100 read storms — both solved with ZINCRBY and short TTL cache.
Key points
- Redis sorted set core. ZADD/ZINCRBY/ZREVRANK/ZREVRANGE — all O(log N).
- ZINCRBY not read-modify-write. Atomic increment — no race on concurrent match results.
- Cassandra durability. Redis is serving layer; rebuild ZSET from Cassandra on failure.
- Top-100 cache. Cache ZREVRANGE result 1s TTL — tournament end storm.
- Multiple boards. Global, regional, friends, seasonal — separate ZSETs updated in pipeline.
- Rank around me. ZREVRANK then ZREVRANGE R-5 R+5.
- Server-side validation. Anti-cheat: validate score server-side before ZINCRBY.
Tradeoffs
Deep dives
At 100M players RANK() OVER scans entire table. ZREVRANK is ~27 comparisons. Top-100 is O(100) regardless of N
Oversimplify why sorted set beats sql — name one component, skip failure modes and metrics.
At 100M players RANK() OVER scans entire table. ZREVRANK is ~27 comparisons. Top-100 is O(100) regardless of N
Name metric + revisit trigger when they push depth.
Redis AOF + RDB. On total loss: batch ZADD from Cassandra — minutes for 100M players. Serve stale cached top-100 during rebuild
Oversimplify durability and rebuild — name one component, skip failure modes and metrics.
Redis AOF + RDB. On total loss: batch ZADD from Cassandra — minutes for 100M players. Serve stale cached top-100 during rebuild
Name metric + revisit trigger when they push depth.
167K ZINCRBY/s within Redis capacity. Top-100 cache with 1s TTL collapses read storm to 1 ZREVRANGE/s
Oversimplify write throughput and tournament storms — name one component, skip failure modes and metrics.
167K ZINCRBY/s within Redis capacity. Top-100 cache with 1s TTL collapses read storm to 1 ZREVRANGE/s
Name metric + revisit trigger when they push depth.
RENAME leaderboard:daily to leaderboard:daily:yesterday atomically at midnight. New empty set for new period
Oversimplify seasonal resets without downtime — name one component, skip failure modes and metrics.
RENAME leaderboard:daily to leaderboard:daily:yesterday atomically at midnight. New empty set for new period
Name metric + revisit trigger when they push depth.
Interview script
Whiteboard
Match -> Score Service -> ZINCRBY leaderboard
|-> Cassandra (audit)
API -> ZREVRANGE top-100 (cached)
API -> ZREVRANK player_idOne diagram: write path ZINCRBY, read paths ZREVRANGE and ZREVRANK.
Design S3-like object storage: PUT/GET/DELETE objects, 11-nines durability, unlimited scale, presigned URLs.
Hard parts: metadata scale, rebalancing on node failure, and large object uploads.
Petabyte payloads and metadata billions of keys — hashing and tiering dominate.
- Metadata vs data separation. Small metadata in SQL/Cassandra. Payload on commodity disks.
- Consistent hashing. Ring with virtual nodes. Add/remove nodes with minimal reshuffle.
- Replication. 3 replicas across racks/AZs. Quorum write before ACK.
- Erasure coding. Cold/archive tier: 10+4 EC reduces storage cost vs 3x replication.
- Multipart upload. Split >100MB into parts. Parallel upload. Commit manifest on complete.
- Presigned URLs. HMAC token lets client upload/download without proxying bytes through API.
Virtual nodes smooth load. On node add: steal ranges. On failure: replicate to successor
Oversimplify consistent hashing and rebalancing — name one component, skip failure modes and metrics.
Virtual nodes smooth load. On node add: steal ranges. On failure: replicate to successor
Name metric + revisit trigger when they push depth.
Sync replicate to 3 AZs before 200 OK on PUT. Background scrub detects bit rot
Oversimplify durability — name one component, skip failure modes and metrics.
Sync replicate to 3 AZs before 200 OK on PUT. Background scrub detects bit rot
Name metric + revisit trigger when they push depth.
Multipart with part ETags. Coordinator commits manifest atomically
Oversimplify large objects — name one component, skip failure modes and metrics.
Multipart with part ETags. Coordinator commits manifest atomically
Name metric + revisit trigger when they push depth.
Prefix index per bucket shard. Paginate with continuation tokens
Oversimplify listing at scale — name one component, skip failure modes and metrics.
Prefix index per bucket shard. Paginate with continuation tokens
Name metric + revisit trigger when they push depth.
Client -> API -> Metadata DB (bucket/key -> node list)
-> Data nodes (replicated chunks)Metadata is the control plane; data nodes are the data plane.
Problem
Design S3-like object storage: PUT/GET/DELETE objects, 11-nines durability, unlimited scale, presigned URLs.
Hard parts: metadata scale, rebalancing on node failure, and large object uploads.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
Petabyte payloads and metadata billions of keys — hashing and tiering dominate.
Key points
- Metadata vs data separation. Small metadata in SQL/Cassandra. Payload on commodity disks.
- Consistent hashing. Ring with virtual nodes. Add/remove nodes with minimal reshuffle.
- Replication. 3 replicas across racks/AZs. Quorum write before ACK.
- Erasure coding. Cold/archive tier: 10+4 EC reduces storage cost vs 3x replication.
- Multipart upload. Split >100MB into parts. Parallel upload. Commit manifest on complete.
- Presigned URLs. HMAC token lets client upload/download without proxying bytes through API.
Tradeoffs
Deep dives
Virtual nodes smooth load. On node add: steal ranges. On failure: replicate to successor
Oversimplify consistent hashing and rebalancing — name one component, skip failure modes and metrics.
Virtual nodes smooth load. On node add: steal ranges. On failure: replicate to successor
Name metric + revisit trigger when they push depth.
Sync replicate to 3 AZs before 200 OK on PUT. Background scrub detects bit rot
Oversimplify durability — name one component, skip failure modes and metrics.
Sync replicate to 3 AZs before 200 OK on PUT. Background scrub detects bit rot
Name metric + revisit trigger when they push depth.
Multipart with part ETags. Coordinator commits manifest atomically
Oversimplify large objects — name one component, skip failure modes and metrics.
Multipart with part ETags. Coordinator commits manifest atomically
Name metric + revisit trigger when they push depth.
Prefix index per bucket shard. Paginate with continuation tokens
Oversimplify listing at scale — name one component, skip failure modes and metrics.
Prefix index per bucket shard. Paginate with continuation tokens
Name metric + revisit trigger when they push depth.
Interview script
Whiteboard
Client -> API -> Metadata DB (bucket/key -> node list)
-> Data nodes (replicated chunks)Metadata is the control plane; data nodes are the data plane.
follower feeds
Photo sharing at massive scale. Two hard problems: media pipeline (ingest → transcode → CDN) and feed generation (hybrid push/pull for celebrity accounts).
The hard part in Instagram is the feed. One new post can create a lot of downstream work, and one feed request can also be expensive if you build it on demand.
There are three main scaling pain points. First, feed generation has a fan-out problem. If you compute the feed at read time, one user request may need posts from hundreds or thousands of followed accounts, then merge and sort them fast. If you precompute feeds at write time, one new post may need to be pushed into millions of follower feeds. Second, media delivery is heavy. Photos and especially videos are large, so uploads, storage, and global low-latency delivery all get expensive fast. Third, load is very uneven. Most users are normal, but celebrity accounts create hot spots because one post can trigger huge write amplification and huge read traffic at the same time.
So the short interview answer is this. Instagram is hard to scale because it combines feed fan-out, massive media storage and delivery, and hot celebrity traffic. That is why a hybrid feed model is usually the best default. Precompute for normal users, then merge celebrity posts at read time.
- Pre-signed URL upload. Client uploads directly to S3. App server only issues the URL and records metadata.
- Async parallel transcode. S3 upload triggers a job. Workers for each resolution run in parallel.
- CDN delivery. Processed images served from CDN. Never serve media from app servers.
- Fan-out on write. Post event → Kafka → fan-out worker → push post_id into follower Redis sorted sets.
- Celebrity threshold. Accounts above a follower threshold get pull-at-read treatment.
accept upload through app servers, store to S3, transcode sequentially before acknowledging
the scaling pain is the read side of media: photos and videos are large, globally distributed, and must load fast. The write side has different constraints: uploads are infrequent, can tolerate latency, but must be resumable
(1) client requests a pre-signed S3 URL — app server never sees the file bytes. (2) Client uploads directly to S3 (bypasses all app servers — critical for cost and throughput). (3) S3 upload event triggers async transcode job (Kafka or Lambda). (4) Transcode workers process in parallel: thumbnail (150×150), feed (1080×1080), full resolution (original), HEVC-compressed video if applicable. (5) Processed outputs written to S3 behind CDN. Transcode workers can be right-sized: CPU-heavy but stateless, easy to scale horizontally. Staff+ failure mode: transcode job fails for one resolution. Store transcode status per (media_id, resolution). Failed resolutions are retried independently. Video is available at successfully-transcoded resolutions while others are processing. User never sees a blank feed card — serve the best available resolution
The fan-out problem is the defining system design challenge for Instagram. At 2B users with 500M DAU, a celebrity with 50M followers posting once generates 50M Redis write operations for that post
async workers handle it
tiered fan-out with explicit threshold. Normal accounts (< 1M followers): fan-out on write — post ID pushed to each follower's Redis sorted set via async Kafka workers. Celebrity accounts (> 1M followers): skip fan-out entirely. At feed read time: user's pre-built feed from Redis + latest N posts from each followed celebrity fetched from Cassandra, merged in-memory
the threshold is not binary — it's a function of follower count and current fan-out worker lag. Monitor lag: if fan-out workers fall behind (>2 min), dynamically lower the threshold to reduce load. The merge at read time: user follows 3 celebrities, each contributes last 20 posts = 60 post candidates + 980 from pre-built feed. Sort by timestamp, serve top 50. Merge cost: O(C × log C) where C is typically < 10, negligible
put one CDN in front of S3, set Cache-Control headers, let it fill naturally
99%+ of media reads are served from CDN — this is what makes Instagram's media delivery economically viable
CDN design: (1) Multi-CDN — use two CDN providers. Route requests to whichever CDN has lower P95 latency for that region (measured by synthetic probes). CDN failover in seconds. (2) Cache warming — for content from large accounts, pre-push content to CDN POPs (Points of Presence) in relevant regions before publication. (3) URL structure encodes resolution: cdn.instagram.com/media/{id}/1080.jpg — CDN can cache all resolutions independently. (4) Signed CDN URLs — prevent hotlinking and unauthorized access. URL includes HMAC signature with expiry. CDN validates signature at edge without origin call. (5) Cache invalidation — when a post is deleted, purge the CDN cache for all resolution variants. CDN APIs support tag-based purge: tag all media variants with the post_id, purge by tag on deletion
Why the deep dives connect to the scaling problem: "Feed fan-out, media blob delivery, and celebrity hot spots." Each deep dive addresses one dimension.
+-------------------+
| Mobile / Web |
| Clients |
+---------+---------+
|
v
+-------------------+
| API Gateway |
| auth rate limit |
+----+----+----+----+
| | |
POST /posts -----+ | +----- GET /feed
POST /follows ---------+
+-------------------+ +-------------------+ +-------------------+
| Post Service | | Follow Service | | Feed Service |
| create post meta | | follow unfollow | | read feed |
+----+---------+----+ +---------+---------+ +----+---------+----+
| | | | |
| | | | |
| v v | v
| +-----------+ +-----------+ | +------------+
| | Posts DB | | FollowsDB | | | Redis |
| | DynamoDB | | DynamoDB | | | feed zset |
| +-----------+ +-----------+ | | post cache |
| | +-----+------+
| | |
| | v
| | +------------+
| +-->| Posts DB |
| | BatchGet |
| +------------+
|
| presigned upload URL
v
+-------------------+ multipart upload +-------------------+
| Blob Storage |<------------------------------>| Client |
| S3 | +-------------------+
+---------+---------+
|
v
+-------------------+
| CDN |
| edge cache media |
+---------+---------+
|
v
+-------------------+
| Media Delivery |
| photos and videos |
+-------------------+
Async fanout path after new post
Post Service
|
v
+-------------------+
| Queue / Topic |
| new post events |
+---------+---------+
|
v
+-------------------+
| Feed Fanout Worker|
| async background |
+----+---------+----+
| |
| v
| +-----------+
| | FollowsDB |
| | followers |
| +-----------+
|
v
+-------------------+
| Redis |
| update feed:user |
+-------------------+
Celebrity hybrid read path
Feed Service
|
+----> Redis precomputed feed for normal accounts
|
+----> Posts DB for recent celebrity posts
|
v
merge by timestamp and return pageThe mental model is two big flows. Write flow stores post metadata and media, then asynchronously updates follower feeds. Read flow pulls mostly from Redis, then hydrates post metadata from the posts store, with a hybrid read for celebrity accounts.
Problem
Photo sharing at massive scale. Two hard problems: media pipeline (ingest → transcode → CDN) and feed generation (hybrid push/pull for celebrity accounts).
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in Instagram is the feed. One new post can create a lot of downstream work, and one feed request can also be expensive if you build it on demand.
There are three main scaling pain points. First, feed generation has a fan-out problem. If you compute the feed at read time, one user request may need posts from hundreds or thousands of followed accounts, then merge and sort them fast. If you precompute feeds at write time, one new post may need to be pushed into millions of follower feeds. Second, media delivery is heavy. Photos and especially videos are large, so uploads, storage, and global low-latency delivery all get expensive fast. Third, load is very uneven. Most users are normal, but celebrity accounts create hot spots because one post can trigger huge write amplification and huge read traffic at the same time.
So the short interview answer is this. Instagram is hard to scale because it combines feed fan-out, massive media storage and delivery, and hot celebrity traffic. That is why a hybrid feed model is usually the best default. Precompute for normal users, then merge celebrity posts at read time.
Key points
- Pre-signed URL upload. Client uploads directly to S3. App server only issues the URL and records metadata.
- Async parallel transcode. S3 upload triggers a job. Workers for each resolution run in parallel.
- CDN delivery. Processed images served from CDN. Never serve media from app servers.
- Fan-out on write. Post event → Kafka → fan-out worker → push post_id into follower Redis sorted sets.
- Celebrity threshold. Accounts above a follower threshold get pull-at-read treatment.
Tradeoffs
Deep dives
accept upload through app servers, store to S3, transcode sequentially before acknowledging
the scaling pain is the read side of media: photos and videos are large, globally distributed, and must load fast. The write side has different constraints: uploads are infrequent, can tolerate latency, but must be resumable
(1) client requests a pre-signed S3 URL — app server never sees the file bytes. (2) Client uploads directly to S3 (bypasses all app servers — critical for cost and throughput). (3) S3 upload event triggers async transcode job (Kafka or Lambda). (4) Transcode workers process in parallel: thumbnail (150×150), feed (1080×1080), full resolution (original), HEVC-compressed video if applicable. (5) Processed outputs written to S3 behind CDN. Transcode workers can be right-sized: CPU-heavy but stateless, easy to scale horizontally. Staff+ failure mode: transcode job fails for one resolution. Store transcode status per (media_id, resolution). Failed resolutions are retried independently. Video is available at successfully-transcoded resolutions while others are processing. User never sees a blank feed card — serve the best available resolution
The fan-out problem is the defining system design challenge for Instagram. At 2B users with 500M DAU, a celebrity with 50M followers posting once generates 50M Redis write operations for that post
async workers handle it
tiered fan-out with explicit threshold. Normal accounts (< 1M followers): fan-out on write — post ID pushed to each follower's Redis sorted set via async Kafka workers. Celebrity accounts (> 1M followers): skip fan-out entirely. At feed read time: user's pre-built feed from Redis + latest N posts from each followed celebrity fetched from Cassandra, merged in-memory
the threshold is not binary — it's a function of follower count and current fan-out worker lag. Monitor lag: if fan-out workers fall behind (>2 min), dynamically lower the threshold to reduce load. The merge at read time: user follows 3 celebrities, each contributes last 20 posts = 60 post candidates + 980 from pre-built feed. Sort by timestamp, serve top 50. Merge cost: O(C × log C) where C is typically < 10, negligible
put one CDN in front of S3, set Cache-Control headers, let it fill naturally
99%+ of media reads are served from CDN — this is what makes Instagram's media delivery economically viable
CDN design: (1) Multi-CDN — use two CDN providers. Route requests to whichever CDN has lower P95 latency for that region (measured by synthetic probes). CDN failover in seconds. (2) Cache warming — for content from large accounts, pre-push content to CDN POPs (Points of Presence) in relevant regions before publication. (3) URL structure encodes resolution: cdn.instagram.com/media/{id}/1080.jpg — CDN can cache all resolutions independently. (4) Signed CDN URLs — prevent hotlinking and unauthorized access. URL includes HMAC signature with expiry. CDN validates signature at edge without origin call. (5) Cache invalidation — when a post is deleted, purge the CDN cache for all resolution variants. CDN APIs support tag-based purge: tag all media variants with the post_id, purge by tag on deletion
Why the deep dives connect to the scaling problem: "Feed fan-out, media blob delivery, and celebrity hot spots." Each deep dive addresses one dimension.
Interview script
Whiteboard
+-------------------+
| Mobile / Web |
| Clients |
+---------+---------+
|
v
+-------------------+
| API Gateway |
| auth rate limit |
+----+----+----+----+
| | |
POST /posts -----+ | +----- GET /feed
POST /follows ---------+
+-------------------+ +-------------------+ +-------------------+
| Post Service | | Follow Service | | Feed Service |
| create post meta | | follow unfollow | | read feed |
+----+---------+----+ +---------+---------+ +----+---------+----+
| | | | |
| | | | |
| v v | v
| +-----------+ +-----------+ | +------------+
| | Posts DB | | FollowsDB | | | Redis |
| | DynamoDB | | DynamoDB | | | feed zset |
| +-----------+ +-----------+ | | post cache |
| | +-----+------+
| | |
| | v
| | +------------+
| +-->| Posts DB |
| | BatchGet |
| +------------+
|
| presigned upload URL
v
+-------------------+ multipart upload +-------------------+
| Blob Storage |<------------------------------>| Client |
| S3 | +-------------------+
+---------+---------+
|
v
+-------------------+
| CDN |
| edge cache media |
+---------+---------+
|
v
+-------------------+
| Media Delivery |
| photos and videos |
+-------------------+
Async fanout path after new post
Post Service
|
v
+-------------------+
| Queue / Topic |
| new post events |
+---------+---------+
|
v
+-------------------+
| Feed Fanout Worker|
| async background |
+----+---------+----+
| |
| v
| +-----------+
| | FollowsDB |
| | followers |
| +-----------+
|
v
+-------------------+
| Redis |
| update feed:user |
+-------------------+
Celebrity hybrid read path
Feed Service
|
+----> Redis precomputed feed for normal accounts
|
+----> Posts DB for recent celebrity posts
|
v
merge by timestamp and return pageThe mental model is two big flows. Write flow stores post metadata and media, then asynchronously updates follower feeds. Read flow pulls mostly from Redis, then hydrates post metadata from the posts store, with a hybrid read for celebrity accounts.
video_id
Count-Min Sketch
Computing and serving the most-viewed videos in real-time at YouTube scale.
The challenge: exact counting at this scale is prohibitively expensive. The system must balance real-time approximation vs exact billing accuracy.
The hard part in YouTube Top K is that it looks like a simple ranking problem, but at scale it becomes a streaming aggregation problem. You are not just storing view counts. You are ingesting a huge firehose of views, updating counts fast enough, and still answering top K queries for different time windows with very low latency.
There are three main scaling pain points. First, write volume is massive. Every view is an event, so naive per view database updates fall over quickly. Second, windowed queries are expensive. Top K for the last hour, day, and month means you cannot just sort one static table. You need rolling aggregates over huge amounts of data. Third, precision plus low latency is a tough combo. If the result must be exact and returned in milliseconds, you usually need precomputation and caching, not on demand scans.
A fourth issue is cardinality. There are billions of videos, but only a tiny fraction belong in the top K. That means you need to process a very large universe of IDs just to find a very small answer set. So the interview summary is this. YouTube Top K is hard because it combines massive write throughput, expensive time window aggregation, and a need to precompute exact rankings fast enough to serve cheaply.
- Lambda Architecture. Real-time path for fast approximate results. Batch path for exact daily reconciliation. Name both paths explicitly.
- Count-Min Sketch. Approximate frequency in O(1) space with 1–5% error. Massive space savings vs exact counting.
- Flink tumbling windows. Aggregate view counts per videoId in fixed time windows.
- Watermarking. Flink watermark with 1-minute allowed lateness. Handles late events.
- Redis sorted set for Top-K. ZINCRBY to update scores, ZREVRANGE to read Top-K.
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
maintain a HashMap of video_id → count, increment on every view event. At 800M videos × 8 bytes = 6.4 GB of counters updated at 11K/sec — too expensive, and one hot video creates a write bottleneck
Count-Min Sketch — a 2D array of W×D counters. On each event: hash video_id with D hash functions, increment the counter at each position. To estimate count: take the minimum of the D values
at 1% error with 99.9% confidence, W ≈ 2718, D ≈ 7. Total memory: 152 KB per time window vs 6.4 GB for exact counting. For Top K specifically: combine CMS with a Min-Heap of K items — only track candidates whose estimated count exceeds the current K-th largest. The heap has K entries; the sketch stays constant size regardless of cardinality
use Flink stream processing only — it handles the volume
stream-only (Kappa architecture) gives approximate results. For ad revenue, creator monetization, and copyright detection, approximate counts are legally insufficient. Lambda Architecture: real-time path gives approximate counts for the trending dashboard (fast, approximate), batch path gives exact counts for billing and reporting (slow, exact)
implementation: real-time path (Flink + CMS + Redis sorted set) runs continuously. Batch path (Spark on S3 event lake) runs daily, produces audited exact counts. Serving layer stores both: consumers use exact counts when available, approximate otherwise. Never use approximate counts for financial reporting
use a single global counter per video — no time window awareness
separate state per window (1hr, 24hr, 7day) using Flink tumbling windows. Each window starts fresh at its boundary
tradeoff: tumbling windows are simpler but the trending list jumps at boundaries — a video popular for the last 59 minutes drops off suddenly at the 1-hour mark. Flink sliding windows give smoother signal but require keeping state for the full window duration. Recommendation: tumbling for longer periods (24h, 7d) where boundary jumps are acceptable, sliding for the 1-hour list where users expect smooth changes. Late event handling: Flink watermark with 2-minute allowed lateness. Events arriving later go to the batch path for exact reconciliation
Why the deep dives connect to the scaling problem: "Massive write throughput, windowed aggregation, and precision vs. latency." Each deep dive addresses one constraint.
+----------------------+
| YouTube Clients |
+----------+-----------+
|
| watch events
v
+----------------------+
| Video Serving System |
+----------+-----------+
|
| publish ViewEvent(videoId, ts)
v
+---------------------+
| Kafka ViewEvent |
| topic partitioned |
| by videoId |
+----+----+----+-----+
| | |
consume | | | consume
v v v
+----------------------------------+
| Flink / Stream Aggregator |
| watermark for late events |
| minute or hour tumbling windows |
| count views per video |
+----------------+-----------------+
|
batched aggregates per shard
|
+------------------------+------------------------+
| | |
v v v
+--------------+ +--------------+ +--------------+
| Views DB S1 | | Views DB S2 | ... | Views DB SN |
| shard by | | shard by | | shard by |
| videoId | | videoId | | videoId |
+------+-------+ +------+-------+ +------+-------+
| | |
| keep window tables | keep window tables |
| | |
| - all_time | - last_hour |
| - last_day | - last_month |
v v v
+---------------------------------------------------------------+
| indexed aggregate tables per shard |
| query top K locally on each shard |
+---------------------------+-----------------------------------+
|
| periodic fanout query
v
+-------------------------------+
| Top K Precompute Job / Cron |
| query each shard for local K |
| merge into global top K |
+---------------+---------------+
|
| write precomputed results
v
+--------------------+
| Redis Cache |
| top-k:last_hour |
| top-k:last_day |
| top-k:last_month |
| top-k:all_time |
+---------+----------+
|
v
+----------------------------+
| Top K API Service |
| GET /views/top-k?window&k |
+-------------+--------------+
|
v
+------------------+
| Load Balancer |
+--------+---------+
|
v
+------------------+
| Clients |
+------------------+If you are presenting this in an interview, the clean story is this. Kafka absorbs the firehose of view events. Flink batches and aggregates views by video for a time bucket. Sharded databases store pre-aggregated counts for each window. A precompute job pulls local top K from each shard, merges them, and writes the final answers into Redis. The API just reads from Redis, which is how you hit the tens of milliseconds latency target.
If you want, I can also give you a simpler interview version with only 6 boxes so it is easier to draw under time pressure.
Problem
Computing and serving the most-viewed videos in real-time at YouTube scale.
The challenge: exact counting at this scale is prohibitively expensive. The system must balance real-time approximation vs exact billing accuracy.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in YouTube Top K is that it looks like a simple ranking problem, but at scale it becomes a streaming aggregation problem. You are not just storing view counts. You are ingesting a huge firehose of views, updating counts fast enough, and still answering top K queries for different time windows with very low latency.
There are three main scaling pain points. First, write volume is massive. Every view is an event, so naive per view database updates fall over quickly. Second, windowed queries are expensive. Top K for the last hour, day, and month means you cannot just sort one static table. You need rolling aggregates over huge amounts of data. Third, precision plus low latency is a tough combo. If the result must be exact and returned in milliseconds, you usually need precomputation and caching, not on demand scans.
A fourth issue is cardinality. There are billions of videos, but only a tiny fraction belong in the top K. That means you need to process a very large universe of IDs just to find a very small answer set. So the interview summary is this. YouTube Top K is hard because it combines massive write throughput, expensive time window aggregation, and a need to precompute exact rankings fast enough to serve cheaply.
Key points
- Lambda Architecture. Real-time path for fast approximate results. Batch path for exact daily reconciliation. Name both paths explicitly.
- Count-Min Sketch. Approximate frequency in O(1) space with 1–5% error. Massive space savings vs exact counting.
- Flink tumbling windows. Aggregate view counts per videoId in fixed time windows.
- Watermarking. Flink watermark with 1-minute allowed lateness. Handles late events.
- Redis sorted set for Top-K. ZINCRBY to update scores, ZREVRANGE to read Top-K.
Tradeoffs
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
maintain a HashMap of video_id → count, increment on every view event. At 800M videos × 8 bytes = 6.4 GB of counters updated at 11K/sec — too expensive, and one hot video creates a write bottleneck
Count-Min Sketch — a 2D array of W×D counters. On each event: hash video_id with D hash functions, increment the counter at each position. To estimate count: take the minimum of the D values
at 1% error with 99.9% confidence, W ≈ 2718, D ≈ 7. Total memory: 152 KB per time window vs 6.4 GB for exact counting. For Top K specifically: combine CMS with a Min-Heap of K items — only track candidates whose estimated count exceeds the current K-th largest. The heap has K entries; the sketch stays constant size regardless of cardinality
use Flink stream processing only — it handles the volume
stream-only (Kappa architecture) gives approximate results. For ad revenue, creator monetization, and copyright detection, approximate counts are legally insufficient. Lambda Architecture: real-time path gives approximate counts for the trending dashboard (fast, approximate), batch path gives exact counts for billing and reporting (slow, exact)
implementation: real-time path (Flink + CMS + Redis sorted set) runs continuously. Batch path (Spark on S3 event lake) runs daily, produces audited exact counts. Serving layer stores both: consumers use exact counts when available, approximate otherwise. Never use approximate counts for financial reporting
use a single global counter per video — no time window awareness
separate state per window (1hr, 24hr, 7day) using Flink tumbling windows. Each window starts fresh at its boundary
tradeoff: tumbling windows are simpler but the trending list jumps at boundaries — a video popular for the last 59 minutes drops off suddenly at the 1-hour mark. Flink sliding windows give smoother signal but require keeping state for the full window duration. Recommendation: tumbling for longer periods (24h, 7d) where boundary jumps are acceptable, sliding for the 1-hour list where users expect smooth changes. Late event handling: Flink watermark with 2-minute allowed lateness. Events arriving later go to the batch path for exact reconciliation
Why the deep dives connect to the scaling problem: "Massive write throughput, windowed aggregation, and precision vs. latency." Each deep dive addresses one constraint.
Interview script
Whiteboard
+----------------------+
| YouTube Clients |
+----------+-----------+
|
| watch events
v
+----------------------+
| Video Serving System |
+----------+-----------+
|
| publish ViewEvent(videoId, ts)
v
+---------------------+
| Kafka ViewEvent |
| topic partitioned |
| by videoId |
+----+----+----+-----+
| | |
consume | | | consume
v v v
+----------------------------------+
| Flink / Stream Aggregator |
| watermark for late events |
| minute or hour tumbling windows |
| count views per video |
+----------------+-----------------+
|
batched aggregates per shard
|
+------------------------+------------------------+
| | |
v v v
+--------------+ +--------------+ +--------------+
| Views DB S1 | | Views DB S2 | ... | Views DB SN |
| shard by | | shard by | | shard by |
| videoId | | videoId | | videoId |
+------+-------+ +------+-------+ +------+-------+
| | |
| keep window tables | keep window tables |
| | |
| - all_time | - last_hour |
| - last_day | - last_month |
v v v
+---------------------------------------------------------------+
| indexed aggregate tables per shard |
| query top K locally on each shard |
+---------------------------+-----------------------------------+
|
| periodic fanout query
v
+-------------------------------+
| Top K Precompute Job / Cron |
| query each shard for local K |
| merge into global top K |
+---------------+---------------+
|
| write precomputed results
v
+--------------------+
| Redis Cache |
| top-k:last_hour |
| top-k:last_day |
| top-k:last_month |
| top-k:all_time |
+---------+----------+
|
v
+----------------------------+
| Top K API Service |
| GET /views/top-k?window&k |
+-------------+--------------+
|
v
+------------------+
| Load Balancer |
+--------+---------+
|
v
+------------------+
| Clients |
+------------------+If you are presenting this in an interview, the clean story is this. Kafka absorbs the firehose of view events. Flink batches and aggregates views by video for a time bucket. Sharded databases store pre-aggregated counts for each window. A precompute job pulls local top K from each shard, merges them, and writes the final answers into Redis. The API just reads from Redis, which is how you hit the tens of milliseconds latency target.
If you want, I can also give you a simpler interview version with only 6 boxes so it is easier to draw under time pressure.
never write to PG
→IN_PROGRESS→DONE
Matching riders with drivers in real time, tracking trips, and calculating pricing dynamically.
The hard parts: driver location is a high-frequency ephemeral stream (never the DB), geospatial matching must be fast, and the trip must be durably recorded.
The hard part in Uber is not storing rides. It is matching the right driver to the right rider very fast while the whole system is constantly changing.
There are three big scaling pain points. First, driver location is a huge real time write stream. If millions of drivers send updates every few seconds, a normal database gets overwhelmed, and proximity search on raw latitude and longitude is too slow. Second, matching needs low latency and strong enough consistency. You cannot assign the same driver to two riders at once, so the system needs some kind of lock or reservation while still moving quickly. Third, demand is bursty and local. A concert ending can create a massive spike in one neighborhood, so even if global traffic looks fine, one region can become a hotspot.
A good interview summary is this. Uber is hard because it combines real time geospatial search, hotspot traffic, and correctness during matching. You are not just finding a nearby driver. You are doing it from fast moving data, under heavy local spikes, without double booking drivers.
- Scope it first. Core: rider requests ride, match to nearby driver, track driver location, complete trip, compute fare. Out of scope unless asked: surge pricing algorithm details, driver incentives, scheduled rides.
- Driver location — Redis GEO only. 2.5M GEOADD writes/sec. Redis handles this; PostgreSQL collapses at ~50K writes/sec. Location is ephemeral: EXPIRE on every key (15s TTL). Expired = driver offline. Never persist raw location to PostgreSQL.
- GEORADIUS for matching. GEORADIUS drivers:available LONGITUDE LATITUDE 5 km ASC COUNT 10. Returns nearest available drivers. Filter by AVAILABLE state (SETNX driver:{id}:status=DISPATCHED — atomic dispatch).
- Trip state machine — PostgreSQL. REQUESTED → MATCHED → DRIVER_EN_ROUTE → IN_PROGRESS → COMPLETED. Each transition is an immutable event. PostgreSQL for ACID durability — the trip is the financial record.
- Surge pricing — geohash cells. Divide city into geohash cells (~1 km²). Compute demand/supply ratio per cell every 2 min. Surge multiplier = f(ratio). ML demand prediction pre-surges 15 min ahead to prevent oscillation.
- Driver state machine — prevent double-dispatch. AVAILABLE → DISPATCHED: SETNX driver:{id}:status = DISPATCHED. First matcher wins. Second matcher finds key exists → pick next driver. TTL on the key prevents stuck state if matcher crashes.
- Failure mode to name. Matching service assigns driver but crashes before confirming to rider: driver gets a ping, rider sees no driver. On client retry, matching service finds driver already DISPATCHED to this trip (idempotency via trip_id) — re-confirms without re-dispatching.
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
write driver location to PostgreSQL on every update — it's the source of truth. At 10M drivers × 1 update/4s = 2.5M writes/sec, PostgreSQL collapses at ~50K writes/sec
Redis GEO (GEOADD) for location — in-memory, O(1) write, supports GEORADIUS queries natively. Set a 15-second TTL on every driver key: if the driver stops sending updates, they're automatically removed from the geo index
never write ephemeral high-frequency data to a relational database. Location is ephemeral — it changes every 4 seconds and is only needed for the current moment. PostgreSQL is for durable trip records, not live location pings. This distinction drives the entire architecture
query Redis for nearby drivers, pick the closest, dispatch
GEORADIUS returns candidates but dispatch requires an atomic state check — the same driver cannot be assigned to two riders simultaneously. SETNX driver:{id}:status = DISPATCHED: first matcher wins, second finds the key already set and picks the next available driver. TTL on the key prevents stuck DISPATCHED state if the matching service crashes mid-dispatch
: matching service assigns a driver but crashes before confirming to the rider. On client retry, the matching service finds the driver already DISPATCHED to this trip (idempotency via trip_id) and re-confirms without re-dispatching. The trip_id is the idempotency key, not the driver_id
store current trip status in Redis for fast access
PostgreSQL for all trip state — REQUESTED → MATCHED → DRIVER_EN_ROUTE → ARRIVED → IN_PROGRESS → COMPLETED. Each transition uses optimistic locking (UPDATE trips SET status=? WHERE id=? AND status=? AND version=N). Zero-row update means a concurrent transition happened — retry or surface the conflict
the trip is the financial record. It must be durable, auditable, and ACID. Event sourcing for the trip: every state transition is an immutable event row. The current state is derived from the event log. This gives a complete audit trail for fare disputes and is required for financial regulatory compliance
Why the deep dives connect to the scaling problem: "Real-time geospatial search, hotspot traffic, correctness during matching." Each deep dive addresses one layer.
+-------------------+
| Rider Client |
| iOS / Android App |
+---------+---------+
|
v
+------+------+
| API Gateway |
| auth, rate |
| limiting |
+------+------+
|
+-------------+-------------+
| |
v v
+-------+--------+ +-------+--------+
| Ride Service | | Notification |
| fares, rides, |<-------->| Service |
| ride state | | push to driver |
+---+--------+---+ +-------+--------+
| | |
| | v
| | +------+------+
| | | Driver |
| | | Client |
| | +------+------+
| | |
| | v
| | PATCH /rides/{id}
| |
| +------------------------------+
| |
v v
+-------+--------+ +---------+---------+
| Ride DB | | Fare DB |
| rides, status, | | estimate records |
| rider, driver | +-------------------+
+----------------+
Fare estimate path
------------------
Rider Client -> API Gateway -> Ride Service -> Maps API
|
v
Fare DB
|
v
Fare response
Matching and location path
--------------------------
+-------------------+ POST /drivers/location +----------------------+
| Driver Client | ---------------------------------> | Location Service |
| GPS updates | | ingest driver coords |
+-------------------+ +----------+-----------+
|
v
+--------+---------+
| Redis Geo Store |
| current driver |
| locations |
+--------+---------+
|
v
+--------+----------+
| Matching Service |
| find nearest |
| available driver |
+---+-----------+---+
| |
| v
| +-----+------+
| | Redis Lock |
| | driver TTL |
| +-----+------+
| |
v v
+------+-----------+------+
| Ride DB update ride |
| requested or accepted |
+-------------------------+
Request ride flow
-----------------Problem
Matching riders with drivers in real time, tracking trips, and calculating pricing dynamically.
The hard parts: driver location is a high-frequency ephemeral stream (never the DB), geospatial matching must be fast, and the trip must be durably recorded.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in Uber is not storing rides. It is matching the right driver to the right rider very fast while the whole system is constantly changing.
There are three big scaling pain points. First, driver location is a huge real time write stream. If millions of drivers send updates every few seconds, a normal database gets overwhelmed, and proximity search on raw latitude and longitude is too slow. Second, matching needs low latency and strong enough consistency. You cannot assign the same driver to two riders at once, so the system needs some kind of lock or reservation while still moving quickly. Third, demand is bursty and local. A concert ending can create a massive spike in one neighborhood, so even if global traffic looks fine, one region can become a hotspot.
A good interview summary is this. Uber is hard because it combines real time geospatial search, hotspot traffic, and correctness during matching. You are not just finding a nearby driver. You are doing it from fast moving data, under heavy local spikes, without double booking drivers.
Key points
- Scope it first. Core: rider requests ride, match to nearby driver, track driver location, complete trip, compute fare. Out of scope unless asked: surge pricing algorithm details, driver incentives, scheduled rides.
- Driver location — Redis GEO only. 2.5M GEOADD writes/sec. Redis handles this; PostgreSQL collapses at ~50K writes/sec. Location is ephemeral: EXPIRE on every key (15s TTL). Expired = driver offline. Never persist raw location to PostgreSQL.
- GEORADIUS for matching. GEORADIUS drivers:available LONGITUDE LATITUDE 5 km ASC COUNT 10. Returns nearest available drivers. Filter by AVAILABLE state (SETNX driver:{id}:status=DISPATCHED — atomic dispatch).
- Trip state machine — PostgreSQL. REQUESTED → MATCHED → DRIVER_EN_ROUTE → IN_PROGRESS → COMPLETED. Each transition is an immutable event. PostgreSQL for ACID durability — the trip is the financial record.
- Surge pricing — geohash cells. Divide city into geohash cells (~1 km²). Compute demand/supply ratio per cell every 2 min. Surge multiplier = f(ratio). ML demand prediction pre-surges 15 min ahead to prevent oscillation.
- Driver state machine — prevent double-dispatch. AVAILABLE → DISPATCHED: SETNX driver:{id}:status = DISPATCHED. First matcher wins. Second matcher finds key exists → pick next driver. TTL on the key prevents stuck state if matcher crashes.
- Failure mode to name. Matching service assigns driver but crashes before confirming to rider: driver gets a ping, rider sees no driver. On client retry, matching service finds driver already DISPATCHED to this trip (idempotency via trip_id) — re-confirms without re-dispatching.
Tradeoffs
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
write driver location to PostgreSQL on every update — it's the source of truth. At 10M drivers × 1 update/4s = 2.5M writes/sec, PostgreSQL collapses at ~50K writes/sec
Redis GEO (GEOADD) for location — in-memory, O(1) write, supports GEORADIUS queries natively. Set a 15-second TTL on every driver key: if the driver stops sending updates, they're automatically removed from the geo index
never write ephemeral high-frequency data to a relational database. Location is ephemeral — it changes every 4 seconds and is only needed for the current moment. PostgreSQL is for durable trip records, not live location pings. This distinction drives the entire architecture
query Redis for nearby drivers, pick the closest, dispatch
GEORADIUS returns candidates but dispatch requires an atomic state check — the same driver cannot be assigned to two riders simultaneously. SETNX driver:{id}:status = DISPATCHED: first matcher wins, second finds the key already set and picks the next available driver. TTL on the key prevents stuck DISPATCHED state if the matching service crashes mid-dispatch
: matching service assigns a driver but crashes before confirming to the rider. On client retry, the matching service finds the driver already DISPATCHED to this trip (idempotency via trip_id) and re-confirms without re-dispatching. The trip_id is the idempotency key, not the driver_id
store current trip status in Redis for fast access
PostgreSQL for all trip state — REQUESTED → MATCHED → DRIVER_EN_ROUTE → ARRIVED → IN_PROGRESS → COMPLETED. Each transition uses optimistic locking (UPDATE trips SET status=? WHERE id=? AND status=? AND version=N). Zero-row update means a concurrent transition happened — retry or surface the conflict
the trip is the financial record. It must be durable, auditable, and ACID. Event sourcing for the trip: every state transition is an immutable event row. The current state is derived from the event log. This gives a complete audit trail for fare disputes and is required for financial regulatory compliance
Why the deep dives connect to the scaling problem: "Real-time geospatial search, hotspot traffic, correctness during matching." Each deep dive addresses one layer.
Interview script
Whiteboard
+-------------------+
| Rider Client |
| iOS / Android App |
+---------+---------+
|
v
+------+------+
| API Gateway |
| auth, rate |
| limiting |
+------+------+
|
+-------------+-------------+
| |
v v
+-------+--------+ +-------+--------+
| Ride Service | | Notification |
| fares, rides, |<-------->| Service |
| ride state | | push to driver |
+---+--------+---+ +-------+--------+
| | |
| | v
| | +------+------+
| | | Driver |
| | | Client |
| | +------+------+
| | |
| | v
| | PATCH /rides/{id}
| |
| +------------------------------+
| |
v v
+-------+--------+ +---------+---------+
| Ride DB | | Fare DB |
| rides, status, | | estimate records |
| rider, driver | +-------------------+
+----------------+
Fare estimate path
------------------
Rider Client -> API Gateway -> Ride Service -> Maps API
|
v
Fare DB
|
v
Fare response
Matching and location path
--------------------------
+-------------------+ POST /drivers/location +----------------------+
| Driver Client | ---------------------------------> | Location Service |
| GPS updates | | ingest driver coords |
+-------------------+ +----------+-----------+
|
v
+--------+---------+
| Redis Geo Store |
| current driver |
| locations |
+--------+---------+
|
v
+--------+----------+
| Matching Service |
| find nearest |
| available driver |
+---+-----------+---+
| |
| v
| +-----+------+
| | Redis Lock |
| | driver TTL |
| +-----+------+
| |
v v
+------+-----------+------+
| Ride DB update ride |
| requested or accepted |
+-------------------------+
Request ride flow
-----------------fast quote reads
+ immutable event log
A commission-free stock trading platform. Real-time market prices, place orders, and portfolio accuracy.
Hard parts: idempotency (retries must not produce double trades), financial integrity (double-entry accounting), and event sourcing for audit trail.
The hard part in Robinhood is that it combines real-time market data with high-stakes correctness. You need prices to update fast, but you also need orders and cancels to be reflected accurately because a stale or lost update can cost real money.
There are three main scaling pain points. First, live price fan-out is big. Many users may watch the same symbol at once, so you do not want every client talking to the exchange directly. You usually centralize exchange connections, ingest the feed once, then fan updates out internally to many app servers. Second, order handling is latency sensitive and consistency sensitive at the same time. A user expects a buy or cancel to happen quickly, but you also need durable local state so you can recover if your system talks to the exchange and then crashes mid-flow. Third, the exchange is an external dependency, which makes everything harder. You have limited connections, limited request patterns, and partial failure cases where your database and the exchange can get out of sync.
A good mental model is this. Robinhood is hard because it is part realtime system and part financial workflow engine. The realtime side is about efficiently broadcasting shared price updates. The harder side is making sure order state stays correct across your system and the exchange, even when failures happen.
- Scope it first. Core: view portfolio, get real-time quotes, place market/limit orders, view order history. Out of scope unless asked: options, margin, crypto, tax lot optimization, fractional shares (unless asked).
- Idempotency key — store before PSP call. Client generates UUID. Server stores idempotency_key (status=PENDING) BEFORE calling broker. On retry: find PENDING key → query broker for status → update to COMPLETED or FAILED. Sequence prevents double orders.
- Event sourcing for ledger. Balance is never stored as a mutable field. Every debit/credit is an immutable event row. Balance = sum(events) materialized by a checkpoint. Required for regulatory compliance — 7-year immutable audit trail.
- Market data — separate pipeline. 10K price updates/sec from exchange → Kafka → Redis. App servers subscribe by ticker. Fan-out to users watching that ticker via WebSocket. High-frequency, approximate-OK (1-2s stale is fine for display).
- Two-phase order execution. (1) Reserve buying power in PG ledger (ACID). (2) Call broker API with idempotency key. (3) On broker success: commit debit + record fill. On broker timeout: mark PENDING, reconcile async. Never debit before broker confirmation.
- Fractional shares — integers only. Store shares as integer units (1 share = 1,000,000 units). All arithmetic stays exact. IEEE 754 float accumulates rounding errors across millions of transactions. Display layer divides by 1M.
- Failure mode to name. Broker API times out — did the order go through? Query broker using the idempotency key before retrying. Idempotency key tells the broker this is a retry, not a new order. Never assume timeout = failure for financial operations.
check if a similar order exists before placing a new one
the hardest financial correctness problem: a network timeout means the client doesn't know if the order went through. If the client retries, it might place two orders. At $50 average trade value and 58 orders/second, a 1% duplicate rate = $29,000/second in double-charges. Unacceptable
design: client-generated idempotency key (UUID v4) included in every order request. Server logic: (1) check if idempotency_key exists in DB — if yes, return the cached response (no new order created). (2) If no, proceed: insert idempotency_key record + create order in one ACID transaction. On broker API call: include the idempotency_key as the broker's own idempotency parameter (Stripe, Alpaca, etc. support this). If broker returns success + idempotency key matches: return cached success. If broker returns error: return cached error. The idempotency record must be stored BEFORE the broker API call — if stored after, a crash between broker success and DB write results in a duplicate on retry. Timing: store idempotency key as PENDING, call broker, update to COMPLETED/FAILED. On retry: if PENDING state found, check broker directly (re-query by the same idempotency key)
maintain a balance column, UPDATE on every debit and credit
a mutable balance field (UPDATE accounts SET balance = balance - 100 WHERE id=?) fails regulatory requirements: you cannot reconstruct the history of how the balance arrived at its current value. Event sourcing: every financial movement is an immutable append-only event row: (account_id, event_type, amount, currency, timestamp, order_id, description). The current balance is computed as the sum of all events for an account
materializing the balance. Recomputing from the full event log on every balance check would be O(N) per query. Solution: maintain a checkpoint table (account_id, balance_as_of_timestamp) updated periodically (e.g., end of day). For live balance: checkpoint + sum of events since checkpoint = current balance. The checkpoint is always derivable from the event log — it's a cache, not the source of truth. If the checkpoint is wrong: recompute from event log (always possible, always correct). Regulatory requirement: event log must be immutable (no UPDATE or DELETE), retained for 7 years
Exchange feed delivers 10,000 price updates/second for thousands of tickers. Distributing this to 1M users watching various tickers is a fan-out problem
WebSocket to every user
topic-based pub/sub with ticker as the topic. Architecture: Market Data Service subscribes to exchange feed → normalizes updates → publishes to Kafka topics (one per ticker). App servers subscribe to tickers that their connected users are watching. On price update: app server pushes to relevant user WebSocket connections
app server maintains an in-memory map of (ticker → [user_connection_ids]). On price update event: look up connections for that ticker, push to each. This is a local fan-out within one server — no cross-server coordination needed because connections are affined by ticker. Hot tickers (GME, AAPL during earnings): the app server handling those users gets more messages but handles them without cross-server coordination. The Kafka topic for hot tickers may need multiple partitions to distribute ingestion load
Why the deep dives connect to the scaling problem: "Real-time system plus financial workflow engine." Deep dive 1 solves order correctness. Deep dive 2 solves auditability. Deep dive 3 solves market data distribution.
+-------------------+
| Mobile / Web |
| Clients |
+---------+---------+
|
HTTPS for API | SSE for live prices
|
+--------v--------+
| Load Balancer |
| sticky for SSE |
+---+----------+---+
| |
+--------------+ +----------------+
| |
+--------v--------+ +---------v---------+
| Order Service | | Symbol Service |
| create cancel | | SSE subscriptions |
| list orders | | fanout to clients |
+---+---------+---+ +----+----------+---+
| | | |
| | | |
| | subscribe by symbol |
| | | |
| | +-----v----------v-----+
| | | Redis Pub Sub |
| | | channels per symbol |
| | +-----------+----------+
| | |
| | |
| +-----v------------------+ |
| | Order DB | |
| | relational sharded by | |
| | userId | |
| +------------------------+ |
| |
| +------------------------+ |
+-->| ExternalOrderId KV |<-----------------+
| externalOrderId -> | trade lookup
| (orderId, userId) | |
+------------------------+ |
|
+--------v---------+
| Trade Processor |
| consumes exchange|
| trade feed |
+--------+---------+
|
updates price cache |
publishes symbol updates |
updates order state |
|
+-------------v--------------+
| Price Cache |
| latest price per symbol |
+-------------+--------------+
|
initial snapshot for SSE
|
|
outbound requests through small set of IPs
|
+-------v-------+
| NAT Gateway |
| / Egress GW |
+-------+-------+
|
sync place cancel APIs
|
async trade feed / webhook
|
+-------v---------+
| Exchange |
| order API + |
| trade feed |
+-----------------+
+-----------------------------+
| Cleanup Worker |
| scans pending and |
| pending_cancel orders |
| reconciles with exchange |
+-----------------------------+The key idea is that you split the system into two flows. One flow is fast live price distribution through Trade Processor -> Redis -> Symbol Service -> SSE clients. The other flow is consistent order handling through Order Service -> DB first -> Exchange -> reconcile state.
If you draw this in an interview, you should call out three important choices. Use SSE for live prices, use a relational orders database partitioned by userId, and use a small egress layer so you do not open too many direct exchange connections.
Problem
A commission-free stock trading platform. Real-time market prices, place orders, and portfolio accuracy.
Hard parts: idempotency (retries must not produce double trades), financial integrity (double-entry accounting), and event sourcing for audit trail.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in Robinhood is that it combines real-time market data with high-stakes correctness. You need prices to update fast, but you also need orders and cancels to be reflected accurately because a stale or lost update can cost real money.
There are three main scaling pain points. First, live price fan-out is big. Many users may watch the same symbol at once, so you do not want every client talking to the exchange directly. You usually centralize exchange connections, ingest the feed once, then fan updates out internally to many app servers. Second, order handling is latency sensitive and consistency sensitive at the same time. A user expects a buy or cancel to happen quickly, but you also need durable local state so you can recover if your system talks to the exchange and then crashes mid-flow. Third, the exchange is an external dependency, which makes everything harder. You have limited connections, limited request patterns, and partial failure cases where your database and the exchange can get out of sync.
A good mental model is this. Robinhood is hard because it is part realtime system and part financial workflow engine. The realtime side is about efficiently broadcasting shared price updates. The harder side is making sure order state stays correct across your system and the exchange, even when failures happen.
Key points
- Scope it first. Core: view portfolio, get real-time quotes, place market/limit orders, view order history. Out of scope unless asked: options, margin, crypto, tax lot optimization, fractional shares (unless asked).
- Idempotency key — store before PSP call. Client generates UUID. Server stores idempotency_key (status=PENDING) BEFORE calling broker. On retry: find PENDING key → query broker for status → update to COMPLETED or FAILED. Sequence prevents double orders.
- Event sourcing for ledger. Balance is never stored as a mutable field. Every debit/credit is an immutable event row. Balance = sum(events) materialized by a checkpoint. Required for regulatory compliance — 7-year immutable audit trail.
- Market data — separate pipeline. 10K price updates/sec from exchange → Kafka → Redis. App servers subscribe by ticker. Fan-out to users watching that ticker via WebSocket. High-frequency, approximate-OK (1-2s stale is fine for display).
- Two-phase order execution. (1) Reserve buying power in PG ledger (ACID). (2) Call broker API with idempotency key. (3) On broker success: commit debit + record fill. On broker timeout: mark PENDING, reconcile async. Never debit before broker confirmation.
- Fractional shares — integers only. Store shares as integer units (1 share = 1,000,000 units). All arithmetic stays exact. IEEE 754 float accumulates rounding errors across millions of transactions. Display layer divides by 1M.
- Failure mode to name. Broker API times out — did the order go through? Query broker using the idempotency key before retrying. Idempotency key tells the broker this is a retry, not a new order. Never assume timeout = failure for financial operations.
Tradeoffs
Deep dives
check if a similar order exists before placing a new one
the hardest financial correctness problem: a network timeout means the client doesn't know if the order went through. If the client retries, it might place two orders. At $50 average trade value and 58 orders/second, a 1% duplicate rate = $29,000/second in double-charges. Unacceptable
design: client-generated idempotency key (UUID v4) included in every order request. Server logic: (1) check if idempotency_key exists in DB — if yes, return the cached response (no new order created). (2) If no, proceed: insert idempotency_key record + create order in one ACID transaction. On broker API call: include the idempotency_key as the broker's own idempotency parameter (Stripe, Alpaca, etc. support this). If broker returns success + idempotency key matches: return cached success. If broker returns error: return cached error. The idempotency record must be stored BEFORE the broker API call — if stored after, a crash between broker success and DB write results in a duplicate on retry. Timing: store idempotency key as PENDING, call broker, update to COMPLETED/FAILED. On retry: if PENDING state found, check broker directly (re-query by the same idempotency key)
maintain a balance column, UPDATE on every debit and credit
a mutable balance field (UPDATE accounts SET balance = balance - 100 WHERE id=?) fails regulatory requirements: you cannot reconstruct the history of how the balance arrived at its current value. Event sourcing: every financial movement is an immutable append-only event row: (account_id, event_type, amount, currency, timestamp, order_id, description). The current balance is computed as the sum of all events for an account
materializing the balance. Recomputing from the full event log on every balance check would be O(N) per query. Solution: maintain a checkpoint table (account_id, balance_as_of_timestamp) updated periodically (e.g., end of day). For live balance: checkpoint + sum of events since checkpoint = current balance. The checkpoint is always derivable from the event log — it's a cache, not the source of truth. If the checkpoint is wrong: recompute from event log (always possible, always correct). Regulatory requirement: event log must be immutable (no UPDATE or DELETE), retained for 7 years
Exchange feed delivers 10,000 price updates/second for thousands of tickers. Distributing this to 1M users watching various tickers is a fan-out problem
WebSocket to every user
topic-based pub/sub with ticker as the topic. Architecture: Market Data Service subscribes to exchange feed → normalizes updates → publishes to Kafka topics (one per ticker). App servers subscribe to tickers that their connected users are watching. On price update: app server pushes to relevant user WebSocket connections
app server maintains an in-memory map of (ticker → [user_connection_ids]). On price update event: look up connections for that ticker, push to each. This is a local fan-out within one server — no cross-server coordination needed because connections are affined by ticker. Hot tickers (GME, AAPL during earnings): the app server handling those users gets more messages but handles them without cross-server coordination. The Kafka topic for hot tickers may need multiple partitions to distribute ingestion load
Why the deep dives connect to the scaling problem: "Real-time system plus financial workflow engine." Deep dive 1 solves order correctness. Deep dive 2 solves auditability. Deep dive 3 solves market data distribution.
Interview script
Whiteboard
+-------------------+
| Mobile / Web |
| Clients |
+---------+---------+
|
HTTPS for API | SSE for live prices
|
+--------v--------+
| Load Balancer |
| sticky for SSE |
+---+----------+---+
| |
+--------------+ +----------------+
| |
+--------v--------+ +---------v---------+
| Order Service | | Symbol Service |
| create cancel | | SSE subscriptions |
| list orders | | fanout to clients |
+---+---------+---+ +----+----------+---+
| | | |
| | | |
| | subscribe by symbol |
| | | |
| | +-----v----------v-----+
| | | Redis Pub Sub |
| | | channels per symbol |
| | +-----------+----------+
| | |
| | |
| +-----v------------------+ |
| | Order DB | |
| | relational sharded by | |
| | userId | |
| +------------------------+ |
| |
| +------------------------+ |
+-->| ExternalOrderId KV |<-----------------+
| externalOrderId -> | trade lookup
| (orderId, userId) | |
+------------------------+ |
|
+--------v---------+
| Trade Processor |
| consumes exchange|
| trade feed |
+--------+---------+
|
updates price cache |
publishes symbol updates |
updates order state |
|
+-------------v--------------+
| Price Cache |
| latest price per symbol |
+-------------+--------------+
|
initial snapshot for SSE
|
|
outbound requests through small set of IPs
|
+-------v-------+
| NAT Gateway |
| / Egress GW |
+-------+-------+
|
sync place cancel APIs
|
async trade feed / webhook
|
+-------v---------+
| Exchange |
| order API + |
| trade feed |
+-----------------+
+-----------------------------+
| Cleanup Worker |
| scans pending and |
| pending_cancel orders |
| reconciles with exchange |
+-----------------------------+The key idea is that you split the system into two flows. One flow is fast live price distribution through Trade Processor -> Redis -> Symbol Service -> SSE clients. The other flow is consistent order handling through Order Service -> DB first -> Exchange -> reconcile state.
If you draw this in an interview, you should call out three important choices. Use SSE for live prices, use a relational orders database partitioned by userId, and use a small egress layer so you do not open too many direct exchange connections.
transform conflicts
Real-time collaborative document editing where multiple users can type simultaneously and all see a consistent document.
The hard part: two users editing the same position concurrently — without OT, the document diverges.
The hard part in Google Docs is concurrent editing on the same shared document. You are not just storing text. You are merging many tiny edits from different users, keeping everyone’s screen nearly in sync, and making sure the document is still correct after races and reconnects.
There are three main scaling pain points. First, consistency is tricky because two users can edit the same spot at the same time, so naive last write wins will lose data or corrupt positions. Second, the system is real time and stateful. Each active document has connected editors, cursor positions, and a stream of low latency updates, which is much harder than normal stateless HTTP traffic. Third, the hot spot is per document. Most docs are quiet, but one shared doc can suddenly have many active editors all sending and receiving updates at once, so you need to route everyone for that doc to the right place and recover cleanly if that server fails.
A fourth issue is storage shape. If you store every keystroke forever, loading a document gets slower and storage keeps growing, so you usually compact old edits into snapshots. The short interview answer is this. Google Docs is hard because it combines concurrent write correctness, real time fan out, and per document hotspot state in one system.
- Scope it first. Core: real-time collaborative text editing with conflict resolution, persistent document storage, version history. Out of scope unless asked: comments, suggestions, offline mode, access control, spreadsheets.
- OT server serializes all ops. Every edit = (type, position, content, version). OT server transforms concurrent ops so all clients converge to the same document. transform(op_A, op_B) → op_A_prime that accounts for op_B having happened first.
- Optimistic local application. Client applies op immediately (low latency UX). When server sends the transformed op back, client reconciles. Users see their own keystrokes instantly — server confirms within ~50ms.
- Op log + snapshots = bounded load time. Every op is an immutable append to Cassandra. Snapshot every 100 ops. On document load: fetch latest snapshot + ops since snapshot. Load time is O(recent ops), not O(total history).
- Document affinity — one OT server per doc. All editors for a document must route to the same OT server (consistent hash by doc_id). OT cannot be distributed across servers for the same document without cross-server ordering — never shard one doc's ops.
- CRDT as the alternative. Commutative data structures — no central server needed. Better for offline-first. Higher per-character memory overhead. Yjs (YATA algorithm) is the production CRDT for rich text. OT is simpler for server-authoritative systems.
- Failure mode to name. OT server crashes mid-session: all clients disconnect. Warm standby (replicated op log from Cassandra) promotes in <10s. Clients reconnect, send buffered ops from last acknowledged version. Op log is always the source of truth.
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
last-write-wins — the most recent save overwrites earlier concurrent edits. Data loss is guaranteed for any concurrent editing session
Operational Transform. Every edit is an operation with type, position, content, and the document version the client was on when they made the edit. The OT server serializes all operations: when concurrent ops arrive, it transforms each op against the ops that happened since the client's version
OT vs CRDT. OT requires a central server to serialize and transform — strong consistency, simpler for rich text. CRDT: commutative operations, no central server needed, better for offline-first. Yjs (YATA algorithm) is the production CRDT for rich text. OT is correct for server-authoritative systems; CRDT is correct for peer-to-peer. State this tradeoff explicitly — it's the question interviewers ask
store every op in Cassandra, replay all ops on document load. For a document with 1M keystrokes, load time = time to replay 1M ops — unbounded and growing forever
periodic snapshots. Every 100 ops: write a full document snapshot to PostgreSQL. On load: fetch latest snapshot + ops since the snapshot. Load time is O(recent ops), not O(total history)
implementation detail: the snapshot interval is a tunable config, not a code constant. Large documents (100-page reports) need more frequent snapshots; small documents can go longer. The snapshot is always derivable from the op log — it's a cache, never the source of truth. If a snapshot is corrupted, you can always reconstruct from the full op log
shard the OT server horizontally — split documents across multiple OT servers for throughput
document affinity is mandatory. All ops for a single document must go to one OT server — distributed sharding across multiple servers for the same document requires cross-shard op ordering, which is the same problem OT was designed to solve. Consistent hash by doc_id routes all editors to the same server
hot document handling: monitor concurrent_editors per document. When >50 concurrent editors: provision a dedicated OT server instance for that document, isolated from other docs. Primary + warm standby per hot document: standby replicates the op log from Cassandra and can promote in <10 seconds on primary failure. This gives hot document HA without sharding the OT logic
Why the deep dives connect to the scaling problem: "Concurrent write correctness, real-time stateful system, per-document hot spot." Each deep dive addresses one layer.
+------------------+
| Client |
| Web / Mobile |
+---------+--------+
|
HTTP + WebSocket
|
+--------v--------+
| API Gateway |
+--------+--------+
|
+----------------+----------------+
| |
| POST /docs | WS /docs/{docId}
| |
+--------v--------+ +--------v-------------------+
| Document Meta | | Document Service |
| Service | | owns active doc sessions |
+--------+--------+ | runs OT transform |
| | tracks presence in memory |
| +----+-------------------+---+
| | |
| | |
+--------v--------+ | |
| Postgres | | |
| Document MetaDB | | |
| docId, title, | | |
| versionId | | |
+-----------------+ | |
| |
append ops | | broadcast edits
| | and cursors
+-------v--------+ |
| Document Ops |<---------+
| DB Cassandra |
| partition by |
| documentId |
+----------------+
In memory inside Document Service per active document
documentId
-> active websocket connections
-> latest loaded operations or materialized doc state
-> pending unacked edits
-> cursor positions
-> presence listThe core idea is simple. Document Meta Service creates documents and stores lightweight metadata. Document Service handles live collaboration, receives edit operations over WebSocket, applies Operational Transformation, writes durable ops to Cassandra, then pushes the transformed updates to every connected editor.
If you want the scaled version, add this around Document Service. ``` Clients | v +------------------+ | Load Balancer | +--------+---------+ | v +-----------------------------------------------+ | Document Service Cluster | | | | +-----------+ +-----------+ +-----------+ | | | Doc Srv A | | Doc Srv B | | Doc Srv C | | | +-----+-----+ +-----+-----+ +-----+-----+ | | \\ | // | | \\ | // | | +---------- v -----------+ | | | Consistent Hash Ring | | | | docId -> owning server | | | +-----------+------------+ | +----------------------+------------------------+ | v +------+------+ | ZooKeeper | | ring config | +-------------+
Each docId maps to one owning Document Service. All editors for the same document connect to the same server. That keeps OT and fanout simple. ``` If you want, I can also give you a cleaner interview-ready version with just 6 boxes so it is easier to draw under time pressure.
Problem
Real-time collaborative document editing where multiple users can type simultaneously and all see a consistent document.
The hard part: two users editing the same position concurrently — without OT, the document diverges.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in Google Docs is concurrent editing on the same shared document. You are not just storing text. You are merging many tiny edits from different users, keeping everyone’s screen nearly in sync, and making sure the document is still correct after races and reconnects.
There are three main scaling pain points. First, consistency is tricky because two users can edit the same spot at the same time, so naive last write wins will lose data or corrupt positions. Second, the system is real time and stateful. Each active document has connected editors, cursor positions, and a stream of low latency updates, which is much harder than normal stateless HTTP traffic. Third, the hot spot is per document. Most docs are quiet, but one shared doc can suddenly have many active editors all sending and receiving updates at once, so you need to route everyone for that doc to the right place and recover cleanly if that server fails.
A fourth issue is storage shape. If you store every keystroke forever, loading a document gets slower and storage keeps growing, so you usually compact old edits into snapshots. The short interview answer is this. Google Docs is hard because it combines concurrent write correctness, real time fan out, and per document hotspot state in one system.
Key points
- Scope it first. Core: real-time collaborative text editing with conflict resolution, persistent document storage, version history. Out of scope unless asked: comments, suggestions, offline mode, access control, spreadsheets.
- OT server serializes all ops. Every edit = (type, position, content, version). OT server transforms concurrent ops so all clients converge to the same document. transform(op_A, op_B) → op_A_prime that accounts for op_B having happened first.
- Optimistic local application. Client applies op immediately (low latency UX). When server sends the transformed op back, client reconciles. Users see their own keystrokes instantly — server confirms within ~50ms.
- Op log + snapshots = bounded load time. Every op is an immutable append to Cassandra. Snapshot every 100 ops. On document load: fetch latest snapshot + ops since snapshot. Load time is O(recent ops), not O(total history).
- Document affinity — one OT server per doc. All editors for a document must route to the same OT server (consistent hash by doc_id). OT cannot be distributed across servers for the same document without cross-server ordering — never shard one doc's ops.
- CRDT as the alternative. Commutative data structures — no central server needed. Better for offline-first. Higher per-character memory overhead. Yjs (YATA algorithm) is the production CRDT for rich text. OT is simpler for server-authoritative systems.
- Failure mode to name. OT server crashes mid-session: all clients disconnect. Warm standby (replicated op log from Cassandra) promotes in <10s. Clients reconnect, send buffered ops from last acknowledged version. Op log is always the source of truth.
Tradeoffs
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
last-write-wins — the most recent save overwrites earlier concurrent edits. Data loss is guaranteed for any concurrent editing session
Operational Transform. Every edit is an operation with type, position, content, and the document version the client was on when they made the edit. The OT server serializes all operations: when concurrent ops arrive, it transforms each op against the ops that happened since the client's version
OT vs CRDT. OT requires a central server to serialize and transform — strong consistency, simpler for rich text. CRDT: commutative operations, no central server needed, better for offline-first. Yjs (YATA algorithm) is the production CRDT for rich text. OT is correct for server-authoritative systems; CRDT is correct for peer-to-peer. State this tradeoff explicitly — it's the question interviewers ask
store every op in Cassandra, replay all ops on document load. For a document with 1M keystrokes, load time = time to replay 1M ops — unbounded and growing forever
periodic snapshots. Every 100 ops: write a full document snapshot to PostgreSQL. On load: fetch latest snapshot + ops since the snapshot. Load time is O(recent ops), not O(total history)
implementation detail: the snapshot interval is a tunable config, not a code constant. Large documents (100-page reports) need more frequent snapshots; small documents can go longer. The snapshot is always derivable from the op log — it's a cache, never the source of truth. If a snapshot is corrupted, you can always reconstruct from the full op log
shard the OT server horizontally — split documents across multiple OT servers for throughput
document affinity is mandatory. All ops for a single document must go to one OT server — distributed sharding across multiple servers for the same document requires cross-shard op ordering, which is the same problem OT was designed to solve. Consistent hash by doc_id routes all editors to the same server
hot document handling: monitor concurrent_editors per document. When >50 concurrent editors: provision a dedicated OT server instance for that document, isolated from other docs. Primary + warm standby per hot document: standby replicates the op log from Cassandra and can promote in <10 seconds on primary failure. This gives hot document HA without sharding the OT logic
Why the deep dives connect to the scaling problem: "Concurrent write correctness, real-time stateful system, per-document hot spot." Each deep dive addresses one layer.
Interview script
Whiteboard
+------------------+
| Client |
| Web / Mobile |
+---------+--------+
|
HTTP + WebSocket
|
+--------v--------+
| API Gateway |
+--------+--------+
|
+----------------+----------------+
| |
| POST /docs | WS /docs/{docId}
| |
+--------v--------+ +--------v-------------------+
| Document Meta | | Document Service |
| Service | | owns active doc sessions |
+--------+--------+ | runs OT transform |
| | tracks presence in memory |
| +----+-------------------+---+
| | |
| | |
+--------v--------+ | |
| Postgres | | |
| Document MetaDB | | |
| docId, title, | | |
| versionId | | |
+-----------------+ | |
| |
append ops | | broadcast edits
| | and cursors
+-------v--------+ |
| Document Ops |<---------+
| DB Cassandra |
| partition by |
| documentId |
+----------------+
In memory inside Document Service per active document
documentId
-> active websocket connections
-> latest loaded operations or materialized doc state
-> pending unacked edits
-> cursor positions
-> presence listThe core idea is simple. Document Meta Service creates documents and stores lightweight metadata. Document Service handles live collaboration, receives edit operations over WebSocket, applies Operational Transformation, writes durable ops to Cassandra, then pushes the transformed updates to every connected editor.
If you want the scaled version, add this around Document Service. ``` Clients | v +------------------+ | Load Balancer | +--------+---------+ | v +-----------------------------------------------+ | Document Service Cluster | | | | +-----------+ +-----------+ +-----------+ | | | Doc Srv A | | Doc Srv B | | Doc Srv C | | | +-----+-----+ +-----+-----+ +-----+-----+ | | \\ | // | | \\ | // | | +---------- v -----------+ | | | Consistent Hash Ring | | | | docId -> owning server | | | +-----------+------------+ | +----------------------+------------------------+ | v +------+------+ | ZooKeeper | | ring config | +-------------+
Each docId maps to one owning Document Service. All editors for the same document connect to the same server. That keeps OT and fanout simple. ``` If you want, I can also give you a cleaner interview-ready version with just 6 boxes so it is easier to draw under time pressure.
stable remapping
Gossip health check
Building a distributed in-memory cache that supports GET/SET, handles node failure gracefully, evicts LRU items under memory pressure, and rebalances efficiently when nodes change.
The hard part is that a distributed cache stops being just a fast in memory map and becomes a coordination problem across many machines.
There are three big scaling pain points. First, you need to shard data across nodes so each key lands on the right machine, and adding or removing nodes should not force you to reshuffle almost everything. That is why consistent hashing matters. Second, you usually want high availability, which means replicas, and replicas create sync problems because reads can become stale and failover gets tricky. Third, hot keys break the nice even distribution. One popular key can overload a single shard even if the rest of the cluster is idle.
A fourth issue is that network cost starts to matter. On one machine, a hash lookup is tiny. In a distributed cache, every get and set may involve a network hop, connection management, and sometimes cross node coordination. So the short interview answer is that distributed cache is hard to scale because you need to keep latency low while handling sharding, replication, rebalancing, and hot spots at the same time.
- Scope it first. Core: GET/SET with TTL, LRU eviction, consistent hashing across nodes, replication for HA. Out of scope unless asked: persistence to disk, Pub/Sub, Lua scripting, sorted sets.
- Consistent hashing — non-negotiable. Modulo hashing: adding one node remaps all K keys → simultaneous cache miss on every key → thundering herd → database crash. Consistent hashing remaps only K/N keys (~10%). This is not an optimization — it is required for safe topology changes.
- Virtual nodes for even distribution. Without vNodes, random ring positions give 3× variance in load across nodes. 150 vNodes per physical node: each node gets 150 ring segments. Adding a new node takes small slices from many existing nodes — balanced from day one.
- LRU = DLL + HashMap, O(1) both. Doubly-linked list for recency order (head = MRU, tail = LRU). HashMap for O(1) key → node lookup. get: HashMap lookup + move node to head. evict: remove tail node + delete from HashMap. Both O(1), no approximation needed.
- Async replication for HA. Primary + 1-2 replicas per shard. Async replication: lower write latency, potential loss of last few writes on primary failure. For a cache (data rebuildable from DB), this tradeoff is correct. Sync replication doubles write latency — wrong for cache.
- Hot key handling. One key accessed 1M/sec overloads one shard. Two layers: (1) local in-process L1 cache on app server (100ms TTL, LRU of top 100 keys — zero network), (2) read replicas for detected hot keys. Hot key detection: monitor per-key QPS, alert at >10K/sec.
- Failure mode to name. Node failure during topology change: consistent hashing means only K/N keys are affected. Those keys miss to the DB (thundering herd risk). Circuit breaker: limit DB fallback rate to DB's sustainable write rate. Replica promotes in <30s via Sentinel.
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
use modulo hashing — key maps to node hash(key) % N. Simple. Adding node N+1 remaps N/(N+1) ≈ 100% of keys simultaneously. Every key misses, every miss hits the database, the database crashes
consistent hashing — place each node at one or more points on a 0→2^32 ring. A key maps to the first node clockwise from its hash. Adding a node: it takes over the key range between itself and its predecessor. Only K/N keys (~10%) are remapped
without virtual nodes, random ring positions create 3× variance in load across physical nodes — one node gets 30% of keys, another gets 5%. 150 virtual nodes per physical node (each node occupies 150 ring positions) gives uniform distribution. Adding a new node takes small slices from many existing nodes simultaneously, keeping load balanced from day one
on every get, scan all entries to find the least recently used one to evict. O(N) eviction — unacceptable at any meaningful cache size
doubly-linked list + HashMap. HashMap stores (key → DLL node) for O(1) lookup. DLL maintains recency order: head = most recently used, tail = least recently used. get: HashMap lookup O(1), move node to head O(1). evict: remove tail O(1), delete from HashMap O(1). put: insert at head O(1), evict tail if at capacity O(1)
implementation: the DLL needs sentinel head and tail nodes to eliminate edge cases (empty list, single element). On get, moving a node from its current position requires unlinking from prev/next and relinking at head — all O(1) pointer operations. LFU as an alternative: tracks access frequency, evicts least frequently used. Better for stable hot/cold workloads but has O(log N) update cost and poor cold-start behavior (new popular key starts at frequency 1, immediately evictable)
replicate synchronously — every write waits for replica ACK before confirming to the client. Correct, but doubles write latency
async replication for a cache. Cache data is derived — it can be rebuilt from the source of truth (the DB). Losing the last few milliseconds of writes on primary failure is acceptable; the data is just re-fetched on the next miss. Primary + 1-2 replicas per shard, async replication, Sentinel-managed automatic failover in <30 seconds
hot key handling: one key accessed 1M times/sec overloads one shard regardless of replication. Two layers: (1) local in-process L1 cache on each app server — top 100 keys, 100ms TTL, zero network hops; (2) read replicas for detected hot keys. Hot key detection: monitor per-key access frequency, alert at >10K QPS. The L1 cache is the most effective lever — it eliminates the hot key problem entirely for the highest-traffic keys
Why the deep dives connect to the scaling problem: "Coordination across machines with low latency." Deep dive 1 solves distribution. Deep dive 2 solves eviction. Deep dive 3 solves availability and hot spots.
+-------------------+
| Clients |
| app servers, APIs |
+---------+---------+
|
get set delete |
v
+---------------------------------+
| Cache Client Library / SDK |
| - consistent hash routing |
| - connection pooling |
| - write batching for hot writes |
| - hot key suffix logic |
+-----------+---------------------+
|
routes directly to owning shard node
|
-------------------------------------------------------------
| Distributed Cache Cluster |
| |
| Shard A Shard B Shard C |
| |
| +-----------+ +-----------+ +-----------+ |
| | Primary A |-------> | Replica A | | Replica A2| |
| | async repl| | read copy | | read copy | |
| +-----+-----+ +-----------+ +-----------+ |
| | |
| | in memory per node |
| v |
| +-------------------------------+ |
| | Hash map key -> node pointer | |
| | Doubly linked list for LRU | |
| | TTL expiry on entries | |
| | Background cleanup process | |
| +-------------------------------+ |
| |
| +-----------+ +-----------+ +-----------+ |
| | Primary B |-------> | Replica B |-------> | Replica B2| |
| +-----+-----+ +-----------+ +-----------+ |
| | |
| v |
| +-------------------------------+ |
| | Hash map + LRU list + TTL | |
| +-------------------------------+ |
| |
| +-----------+ +-----------+ +-----------+ |
| | Primary C |-------> | Replica C |-------> | Replica C2| |
| +-----+-----+ +-----------+ +-----------+ |
| | |
| v |
| +-------------------------------+ |
| | Hash map + LRU list + TTL | |
| +-------------------------------+ |
-------------------------------------------------------------
Hot read handling
hot:key
|
+--> hot:key#1 on Shard A
+--> hot:key#2 on Shard B
+--> hot:key#3 on Shard C
Reads pick one copy to spread load.
Writes update all copies asynchronously.
Hot write handling
counter:item42
|
+--> counter:item42:1 on Shard A
+--> counter:item42:2 on Shard B
+--> counter:item42:3 on Shard C
Writes are spread across suffixes.
Reads aggregate across shards.The main mental model is this. Each cache node is just a fast in memory LRU cache, and the distributed part comes from sharding keys across many nodes with replication for availability. If you want, I can also give you a cleaner interview sized version that fits on one whiteboard.
Problem
Building a distributed in-memory cache that supports GET/SET, handles node failure gracefully, evicts LRU items under memory pressure, and rebalances efficiently when nodes change.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part is that a distributed cache stops being just a fast in memory map and becomes a coordination problem across many machines.
There are three big scaling pain points. First, you need to shard data across nodes so each key lands on the right machine, and adding or removing nodes should not force you to reshuffle almost everything. That is why consistent hashing matters. Second, you usually want high availability, which means replicas, and replicas create sync problems because reads can become stale and failover gets tricky. Third, hot keys break the nice even distribution. One popular key can overload a single shard even if the rest of the cluster is idle.
A fourth issue is that network cost starts to matter. On one machine, a hash lookup is tiny. In a distributed cache, every get and set may involve a network hop, connection management, and sometimes cross node coordination. So the short interview answer is that distributed cache is hard to scale because you need to keep latency low while handling sharding, replication, rebalancing, and hot spots at the same time.
Key points
- Scope it first. Core: GET/SET with TTL, LRU eviction, consistent hashing across nodes, replication for HA. Out of scope unless asked: persistence to disk, Pub/Sub, Lua scripting, sorted sets.
- Consistent hashing — non-negotiable. Modulo hashing: adding one node remaps all K keys → simultaneous cache miss on every key → thundering herd → database crash. Consistent hashing remaps only K/N keys (~10%). This is not an optimization — it is required for safe topology changes.
- Virtual nodes for even distribution. Without vNodes, random ring positions give 3× variance in load across nodes. 150 vNodes per physical node: each node gets 150 ring segments. Adding a new node takes small slices from many existing nodes — balanced from day one.
- LRU = DLL + HashMap, O(1) both. Doubly-linked list for recency order (head = MRU, tail = LRU). HashMap for O(1) key → node lookup. get: HashMap lookup + move node to head. evict: remove tail node + delete from HashMap. Both O(1), no approximation needed.
- Async replication for HA. Primary + 1-2 replicas per shard. Async replication: lower write latency, potential loss of last few writes on primary failure. For a cache (data rebuildable from DB), this tradeoff is correct. Sync replication doubles write latency — wrong for cache.
- Hot key handling. One key accessed 1M/sec overloads one shard. Two layers: (1) local in-process L1 cache on app server (100ms TTL, LRU of top 100 keys — zero network), (2) read replicas for detected hot keys. Hot key detection: monitor per-key QPS, alert at >10K/sec.
- Failure mode to name. Node failure during topology change: consistent hashing means only K/N keys are affected. Those keys miss to the DB (thundering herd risk). Circuit breaker: limit DB fallback rate to DB's sustainable write rate. Replica promotes in <30s via Sentinel.
Tradeoffs
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
use modulo hashing — key maps to node hash(key) % N. Simple. Adding node N+1 remaps N/(N+1) ≈ 100% of keys simultaneously. Every key misses, every miss hits the database, the database crashes
consistent hashing — place each node at one or more points on a 0→2^32 ring. A key maps to the first node clockwise from its hash. Adding a node: it takes over the key range between itself and its predecessor. Only K/N keys (~10%) are remapped
without virtual nodes, random ring positions create 3× variance in load across physical nodes — one node gets 30% of keys, another gets 5%. 150 virtual nodes per physical node (each node occupies 150 ring positions) gives uniform distribution. Adding a new node takes small slices from many existing nodes simultaneously, keeping load balanced from day one
on every get, scan all entries to find the least recently used one to evict. O(N) eviction — unacceptable at any meaningful cache size
doubly-linked list + HashMap. HashMap stores (key → DLL node) for O(1) lookup. DLL maintains recency order: head = most recently used, tail = least recently used. get: HashMap lookup O(1), move node to head O(1). evict: remove tail O(1), delete from HashMap O(1). put: insert at head O(1), evict tail if at capacity O(1)
implementation: the DLL needs sentinel head and tail nodes to eliminate edge cases (empty list, single element). On get, moving a node from its current position requires unlinking from prev/next and relinking at head — all O(1) pointer operations. LFU as an alternative: tracks access frequency, evicts least frequently used. Better for stable hot/cold workloads but has O(log N) update cost and poor cold-start behavior (new popular key starts at frequency 1, immediately evictable)
replicate synchronously — every write waits for replica ACK before confirming to the client. Correct, but doubles write latency
async replication for a cache. Cache data is derived — it can be rebuilt from the source of truth (the DB). Losing the last few milliseconds of writes on primary failure is acceptable; the data is just re-fetched on the next miss. Primary + 1-2 replicas per shard, async replication, Sentinel-managed automatic failover in <30 seconds
hot key handling: one key accessed 1M times/sec overloads one shard regardless of replication. Two layers: (1) local in-process L1 cache on each app server — top 100 keys, 100ms TTL, zero network hops; (2) read replicas for detected hot keys. Hot key detection: monitor per-key access frequency, alert at >10K QPS. The L1 cache is the most effective lever — it eliminates the hot key problem entirely for the highest-traffic keys
Why the deep dives connect to the scaling problem: "Coordination across machines with low latency." Deep dive 1 solves distribution. Deep dive 2 solves eviction. Deep dive 3 solves availability and hot spots.
Interview script
Whiteboard
+-------------------+
| Clients |
| app servers, APIs |
+---------+---------+
|
get set delete |
v
+---------------------------------+
| Cache Client Library / SDK |
| - consistent hash routing |
| - connection pooling |
| - write batching for hot writes |
| - hot key suffix logic |
+-----------+---------------------+
|
routes directly to owning shard node
|
-------------------------------------------------------------
| Distributed Cache Cluster |
| |
| Shard A Shard B Shard C |
| |
| +-----------+ +-----------+ +-----------+ |
| | Primary A |-------> | Replica A | | Replica A2| |
| | async repl| | read copy | | read copy | |
| +-----+-----+ +-----------+ +-----------+ |
| | |
| | in memory per node |
| v |
| +-------------------------------+ |
| | Hash map key -> node pointer | |
| | Doubly linked list for LRU | |
| | TTL expiry on entries | |
| | Background cleanup process | |
| +-------------------------------+ |
| |
| +-----------+ +-----------+ +-----------+ |
| | Primary B |-------> | Replica B |-------> | Replica B2| |
| +-----+-----+ +-----------+ +-----------+ |
| | |
| v |
| +-------------------------------+ |
| | Hash map + LRU list + TTL | |
| +-------------------------------+ |
| |
| +-----------+ +-----------+ +-----------+ |
| | Primary C |-------> | Replica C |-------> | Replica C2| |
| +-----+-----+ +-----------+ +-----------+ |
| | |
| v |
| +-------------------------------+ |
| | Hash map + LRU list + TTL | |
| +-------------------------------+ |
-------------------------------------------------------------
Hot read handling
hot:key
|
+--> hot:key#1 on Shard A
+--> hot:key#2 on Shard B
+--> hot:key#3 on Shard C
Reads pick one copy to spread load.
Writes update all copies asynchronously.
Hot write handling
counter:item42
|
+--> counter:item42:1 on Shard A
+--> counter:item42:2 on Shard B
+--> counter:item42:3 on Shard C
Writes are spread across suffixes.
Reads aggregate across shards.The main mental model is this. Each cache node is just a fast in memory LRU cache, and the distributed part comes from sharding keys across many nodes with replication for availability. If you want, I can also give you a cleaner interview sized version that fits on one whiteboard.
360p/720p/1080p/4K
bitrate stream
Hosting and streaming video at YouTube scale — billions of videos, hundreds of millions of simultaneous viewers.
Two hard problems: ingestion and transcoding (raw video → multiple quality levels → CDN), and adaptive bitrate streaming.
The hard part in YouTube is not storing videos. It is handling huge video files on the upload path and serving smooth playback on the watch path.
There are three main scaling pain points. First, videos are large blobs, so uploads need multipart and resumable transfer straight to blob storage instead of passing through app servers. Second, playback is bandwidth sensitive. Users have different devices and network quality, so you usually split videos into small segments, transcode them into multiple qualities, and let the client switch between them during playback. Third, reads are extremely skewed. A video is uploaded once but may be watched millions of times, so popular videos create hot spots and you need CDN caching for segments and manifests plus caching for metadata.
The extra wrinkle is post processing. One upload turns into a pipeline that splits, transcodes, and writes many output files, which is a lot of CPU work even before anyone watches the video. So the short interview answer is this. YouTube is hard to scale because it combines large file uploads, expensive video processing, and massive read heavy streaming with hot viral traffic.
- TUS resumable upload. Chunked upload protocol. If upload fails, resume from last successful chunk.
- Async parallel transcode. S3 upload triggers a job. Separate workers for each resolution run in parallel. Output: HLS/DASH segments.
- CDN = cost driver. CDN absorbs 99%+ of bandwidth. App servers handle only metadata APIs.
- ABR streaming. HLS manifest lists quality levels. Client measures bandwidth and requests the appropriate tier.
- Elasticsearch for search. Full-text search over titles and descriptions.
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
standard HTTP multipart upload, transcode sequentially through all resolutions. A 10 GB upload that fails at 9.9 GB restarts from zero. Sequential transcode means 360p isn't available until 4K finishes — user waits minutes before their video is watchable
TUS resumable protocol. Client splits the file into 10 MB chunks, tracks each independently. On network failure: resume from the last acknowledged chunk. Parallel transcode: one Flink job per resolution running simultaneously on separate worker instances. 360p is available within seconds of upload completion; 4K follows asynchronously
: transcode job fails for a specific resolution. Store transcode status per (video_id, resolution). Failed resolutions retry independently. Video is available at successful resolutions while others process. Never block video availability on full transcode completion — this is the difference between a 2-minute and a 20-minute time-to-publish
serve one video quality to all users — high quality for everyone, or low quality to save bandwidth
HLS adaptive bitrate streaming. The master manifest (.m3u8) lists all quality variants with bandwidth requirements. Client downloads the master manifest, picks initial quality based on current bandwidth estimate, then switches dynamically per segment
switching logic: switch up if measured bandwidth > 1.3× current bitrate AND buffer > 30 seconds; switch down if measured bandwidth < 0.8× current bitrate OR buffer drops below 10 seconds. This asymmetry (requires more headroom to switch up than to switch down) prevents oscillation — the player doesn't constantly flip between quality tiers on variable connections. Server-side: HLS segments stored in S3 with deterministic URL structure /{video_id}/{resolution}/{segment_number}.ts. The CDN pre-fetches upcoming segments during playback because the next segment URL is predictable
put a CDN in front of S3 and set a long cache TTL
multi-layer CDN strategy. (1) Multi-CDN routing: two providers, DNS routes to the one with lower measured P95 latency for that region. Failover in seconds. (2) Origin shielding: instead of all CDN POPs fetching independently from S3 on a miss, a small set of shield POPs (10-20 globally) fetch from S3 and local POPs fetch from the shield. Reduces S3 request rate from O(POPs × misses) to O(shields × misses)
cache warming: for large channels, the CDN push API pre-populates segments at relevant POPs before the video goes live. No cold cache for the first million viewers of a major release. Purge strategy: deleted or copyright-struck videos need instant CDN purge across all POPs. Tag-based purge: all segments of a video_id are tagged at upload time, purged with a single API call on deletion
Why the deep dives connect to the scaling problem: "Large blob uploads, expensive transcoding, massive read-heavy streaming." Each deep dive addresses one layer.
+-------------------+
| Users |
| uploader, viewer |
+---------+---------+
|
HTTPS |
v
+-------------------+
| Load Balancer |
+---------+---------+
|
v
+-------------------+
| Video Service |
| stateless API |
+----+---------+----+
| |
| |
| +----------------------+
| |
v v
+------------------------+ +---------------------+
| Metadata Cache | | Video Metadata DB |
| distributed cache | | Cassandra |
+------------------------+ +---------------------+
UPLOAD FLOW
===========
1. Client asks for upload session and presigned URL
User ---> Video Service ---> Metadata DB
|
+--> returns videoId, multipart upload info,
presigned URLs
2. Client uploads video directly to blob storage
User ---------------------------------------------> S3 Blob Storage
multipart upload of raw video chunks
3. Client reports chunk progress
User ---> Video Service ---> Metadata DB
chunk uploaded status, ETag info
4. Upload completion triggers processing
S3 ObjectCreated event ---> Processing Orchestrator
PROCESSING PIPELINE
===================
+-------------------------+
| Processing Orchestrator |
| DAG workflow manager |
+-----------+-------------+
|
---------------------------------------------------
| | |
v v v
+----------------+ +----------------+ +----------------+
| Segment Worker | | Audio Worker | | Transcript |
| split raw file | | audio process | | Worker |
+-------+--------+ +--------+-------+ +--------+-------+
| | |
v v v
+----------------+ +----------------+ +----------------+
| Transcode | | audio outputs | | transcript out |
| Workers | | in S3 | | in S3 |
| many in parallel| +----------------+ +----------------+
+-------+--------+
|
v
+------------------------+
| Manifest Generator |
| primary + media files |
+-----------+------------+
|
v
+------------------------+
| S3 processed assets |
| segments, manifests |
+-----------+------------+
|
v
+------------------------+
| Metadata DB update |
| manifest URL, status |
| upload complete |
+------------------------+
PLAYBACK FLOW
=============
User ---> Video Service ---> Cache ---> Metadata DB
|
+--> returns manifest URL and metadata
User ---> CDN ---> S3 processed assets
| manifests and video segments
|
+--> edge serves cached content when possible
Client playback logic
- fetch manifest
- pick bitrate based on network
- download first segment
- keep downloading next segments
- switch quality up or down as bandwidth changes
FULL SYSTEM VIEW
================
+-------------------+
| Users |
+---------+---------+
|
v
+-------------------+
| Load Balancer |
+---------+---------+
|
v
+-------------------+
| Video Service |
+---+-----------+---+
| |
v v
+-------------+ +------------------+
| Cache | | Metadata DB |
| popular md | | Cassandra |
+-------------+ +------------------+
upload session / metadata | ^
| |
v |
+-------------------------------+
| S3 Blob Storage |
| raw uploads |
| processed segments/manifests |
+---------------+---------------+
|
object event |
v
+-------------------------------+
| Processing Orchestrator |
| workflow / DAG manager |
+---------------+---------------+
|
parallel worker fleet
|
-------------------------------------------------
| | |
v v v
+-------------+ +---------------+ +---------------+
| Split | | Transcode | | Other media |
| workers | | workers | | workers |
+-------------+ +---------------+ +---------------+
|
v
+----------------+
| Manifest Gen |
+--------+-------+
|
v
+-----------+
| CDN |
| edge cache|
+-----+-----+
|
v
UsersThe mental model is two big paths. Upload goes client to S3, then processing pipeline, then metadata update. Watch goes client to metadata, then manifest, then CDN segment fetches.
If you want, I can also give you a smaller interview friendly version that fits in 60 seconds on a whiteboard.
Problem
Hosting and streaming video at YouTube scale — billions of videos, hundreds of millions of simultaneous viewers.
Two hard problems: ingestion and transcoding (raw video → multiple quality levels → CDN), and adaptive bitrate streaming.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in YouTube is not storing videos. It is handling huge video files on the upload path and serving smooth playback on the watch path.
There are three main scaling pain points. First, videos are large blobs, so uploads need multipart and resumable transfer straight to blob storage instead of passing through app servers. Second, playback is bandwidth sensitive. Users have different devices and network quality, so you usually split videos into small segments, transcode them into multiple qualities, and let the client switch between them during playback. Third, reads are extremely skewed. A video is uploaded once but may be watched millions of times, so popular videos create hot spots and you need CDN caching for segments and manifests plus caching for metadata.
The extra wrinkle is post processing. One upload turns into a pipeline that splits, transcodes, and writes many output files, which is a lot of CPU work even before anyone watches the video. So the short interview answer is this. YouTube is hard to scale because it combines large file uploads, expensive video processing, and massive read heavy streaming with hot viral traffic.
Key points
- TUS resumable upload. Chunked upload protocol. If upload fails, resume from last successful chunk.
- Async parallel transcode. S3 upload triggers a job. Separate workers for each resolution run in parallel. Output: HLS/DASH segments.
- CDN = cost driver. CDN absorbs 99%+ of bandwidth. App servers handle only metadata APIs.
- ABR streaming. HLS manifest lists quality levels. Client measures bandwidth and requests the appropriate tier.
- Elasticsearch for search. Full-text search over titles and descriptions.
Tradeoffs
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
standard HTTP multipart upload, transcode sequentially through all resolutions. A 10 GB upload that fails at 9.9 GB restarts from zero. Sequential transcode means 360p isn't available until 4K finishes — user waits minutes before their video is watchable
TUS resumable protocol. Client splits the file into 10 MB chunks, tracks each independently. On network failure: resume from the last acknowledged chunk. Parallel transcode: one Flink job per resolution running simultaneously on separate worker instances. 360p is available within seconds of upload completion; 4K follows asynchronously
: transcode job fails for a specific resolution. Store transcode status per (video_id, resolution). Failed resolutions retry independently. Video is available at successful resolutions while others process. Never block video availability on full transcode completion — this is the difference between a 2-minute and a 20-minute time-to-publish
serve one video quality to all users — high quality for everyone, or low quality to save bandwidth
HLS adaptive bitrate streaming. The master manifest (.m3u8) lists all quality variants with bandwidth requirements. Client downloads the master manifest, picks initial quality based on current bandwidth estimate, then switches dynamically per segment
switching logic: switch up if measured bandwidth > 1.3× current bitrate AND buffer > 30 seconds; switch down if measured bandwidth < 0.8× current bitrate OR buffer drops below 10 seconds. This asymmetry (requires more headroom to switch up than to switch down) prevents oscillation — the player doesn't constantly flip between quality tiers on variable connections. Server-side: HLS segments stored in S3 with deterministic URL structure /{video_id}/{resolution}/{segment_number}.ts. The CDN pre-fetches upcoming segments during playback because the next segment URL is predictable
put a CDN in front of S3 and set a long cache TTL
multi-layer CDN strategy. (1) Multi-CDN routing: two providers, DNS routes to the one with lower measured P95 latency for that region. Failover in seconds. (2) Origin shielding: instead of all CDN POPs fetching independently from S3 on a miss, a small set of shield POPs (10-20 globally) fetch from S3 and local POPs fetch from the shield. Reduces S3 request rate from O(POPs × misses) to O(shields × misses)
cache warming: for large channels, the CDN push API pre-populates segments at relevant POPs before the video goes live. No cold cache for the first million viewers of a major release. Purge strategy: deleted or copyright-struck videos need instant CDN purge across all POPs. Tag-based purge: all segments of a video_id are tagged at upload time, purged with a single API call on deletion
Why the deep dives connect to the scaling problem: "Large blob uploads, expensive transcoding, massive read-heavy streaming." Each deep dive addresses one layer.
Interview script
Whiteboard
+-------------------+
| Users |
| uploader, viewer |
+---------+---------+
|
HTTPS |
v
+-------------------+
| Load Balancer |
+---------+---------+
|
v
+-------------------+
| Video Service |
| stateless API |
+----+---------+----+
| |
| |
| +----------------------+
| |
v v
+------------------------+ +---------------------+
| Metadata Cache | | Video Metadata DB |
| distributed cache | | Cassandra |
+------------------------+ +---------------------+
UPLOAD FLOW
===========
1. Client asks for upload session and presigned URL
User ---> Video Service ---> Metadata DB
|
+--> returns videoId, multipart upload info,
presigned URLs
2. Client uploads video directly to blob storage
User ---------------------------------------------> S3 Blob Storage
multipart upload of raw video chunks
3. Client reports chunk progress
User ---> Video Service ---> Metadata DB
chunk uploaded status, ETag info
4. Upload completion triggers processing
S3 ObjectCreated event ---> Processing Orchestrator
PROCESSING PIPELINE
===================
+-------------------------+
| Processing Orchestrator |
| DAG workflow manager |
+-----------+-------------+
|
---------------------------------------------------
| | |
v v v
+----------------+ +----------------+ +----------------+
| Segment Worker | | Audio Worker | | Transcript |
| split raw file | | audio process | | Worker |
+-------+--------+ +--------+-------+ +--------+-------+
| | |
v v v
+----------------+ +----------------+ +----------------+
| Transcode | | audio outputs | | transcript out |
| Workers | | in S3 | | in S3 |
| many in parallel| +----------------+ +----------------+
+-------+--------+
|
v
+------------------------+
| Manifest Generator |
| primary + media files |
+-----------+------------+
|
v
+------------------------+
| S3 processed assets |
| segments, manifests |
+-----------+------------+
|
v
+------------------------+
| Metadata DB update |
| manifest URL, status |
| upload complete |
+------------------------+
PLAYBACK FLOW
=============
User ---> Video Service ---> Cache ---> Metadata DB
|
+--> returns manifest URL and metadata
User ---> CDN ---> S3 processed assets
| manifests and video segments
|
+--> edge serves cached content when possible
Client playback logic
- fetch manifest
- pick bitrate based on network
- download first segment
- keep downloading next segments
- switch quality up or down as bandwidth changes
FULL SYSTEM VIEW
================
+-------------------+
| Users |
+---------+---------+
|
v
+-------------------+
| Load Balancer |
+---------+---------+
|
v
+-------------------+
| Video Service |
+---+-----------+---+
| |
v v
+-------------+ +------------------+
| Cache | | Metadata DB |
| popular md | | Cassandra |
+-------------+ +------------------+
upload session / metadata | ^
| |
v |
+-------------------------------+
| S3 Blob Storage |
| raw uploads |
| processed segments/manifests |
+---------------+---------------+
|
object event |
v
+-------------------------------+
| Processing Orchestrator |
| workflow / DAG manager |
+---------------+---------------+
|
parallel worker fleet
|
-------------------------------------------------
| | |
v v v
+-------------+ +---------------+ +---------------+
| Split | | Transcode | | Other media |
| workers | | workers | | workers |
+-------------+ +---------------+ +---------------+
|
v
+----------------+
| Manifest Gen |
+--------+-------+
|
v
+-----------+
| CDN |
| edge cache|
+-----+-----+
|
v
UsersThe mental model is two big paths. Upload goes client to S3, then processing pipeline, then metadata update. Watch goes client to metadata, then manifest, then CDN segment fetches.
If you want, I can also give you a smaller interview friendly version that fits in 60 seconds on a whiteboard.
per-domain rate limit
robots.txt check
Cassandra → confirm
Systematically discovering and fetching web pages. Handle billions of URLs, respect site rate limits, avoid re-crawling duplicates, and recover from failures.
Hard parts: politeness, deduplication at scale, and prioritizing fresh content.
The hard part in Web Crawler is not storing pages. It is coordinating a huge number of fetches across the public internet without wasting work or being rude to other sites.
There are four scaling pain points you should call out. First, the crawl frontier gets huge. You keep discovering new URLs, but you need to dedupe them so different workers do not crawl the same page again and again. Second, politeness limits parallelism. You may want massive throughput, but you still need per domain rate limits and robots.txt checks, so scaling is not just adding more workers. Third, the internet is unreliable. DNS lookups, slow servers, dead links, retries, and crawler traps all waste time unless you pipeline the work and track progress carefully. Fourth, the workload is very uneven. Some domains are tiny, some are enormous, and some generate endless near-duplicate pages, so load balancing is messy.
A good interview summary is this. Web Crawler is hard because it combines massive frontier management, external bottlenecks like DNS and website limits, duplicate avoidance, and fault tolerance in one system.
- Scope it first. Core: crawl the web, extract text, store pages, enable search indexing. Out of scope unless asked: JavaScript rendering, login-gated content, real-time re-crawl, entity extraction.
- URL frontier is the core data structure. Two-tier priority queue. Back queues: one per domain (enforces politeness — one request per domain per N seconds). Front queue: priority-ordered list of domains to crawl next. Never crawl faster than the site allows.
- Always respect robots.txt. Legal requirement in most jurisdictions. Cache per domain (TTL 24h). Check before every fetch. Violation = IP ban. State this proactively in interviews — it signals production awareness.
- Bloom filter for URL dedup. 96K new URL candidates/sec. DB-only dedup = impossible. Bloom filter (9.6 GB for 5B URLs at 1% FPR): O(1) per check. False positive = occasionally skip a valid URL — acceptable. Cassandra exact check only for Bloom positives.
- DNS caching per domain. At 1,929 fetches/sec, uncached DNS at 100ms per lookup = DNS becomes the bottleneck. Cache per domain with TTL (1 hour). Pre-fetch DNS for domains in the near-term frontier queue. DNS miss rate target < 5%.
- Content dedup — SimHash. 30-40% of the web is near-duplicate content. SimHash fingerprint: 64-bit, Hamming distance < 3 = near-duplicate. Partition fingerprints by first K bits for efficient lookup (LSH). Canonical URL from <link rel=canonical> takes priority.
- Failure mode to name. Crawler enters a trap (infinite URL space): per-domain URL count limit (1M max), path depth limit (10 levels), counter-pattern detection. Without traps, one rogue domain can starve the entire crawl budget.
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
a single queue of URLs to crawl in BFS order — simple and correct for small scale
two-tier priority frontier. Back queues: one per domain, enforces politeness (one request per domain per N seconds, respects robots.txt Crawl-delay). Front queue: priority-ordered list of (domain, priority) pairs. The scheduler picks the highest-priority domain whose next_allowed_crawl_time ≤ now. This ensures politeness and value-maximization simultaneously
priority = f(PageRank estimate, domain authority, freshness score). High-authority domains (CNN, Wikipedia) get crawled most frequently. Pages within a domain are prioritized by inbound link count. The priority function is configurable — it's the lever that determines what percentage of your crawl budget goes to fresh high-value content vs. long-tail pages
store all crawled URLs in a database, check before each fetch. At 96K new URL candidates/sec, 96K DB queries/sec for dedup alone — impossible
Bloom filter as a fast first-pass. At 5B URLs, 1% false positive rate: 9.6 GB memory, 7 hash functions, O(1) per check. False positive = occasionally skipping a valid URL — acceptable tradeoff. Cassandra exact check only for Bloom filter positives (~1% of checks)
operational concern: Bloom filter fills up. Monitor load factor — alert at 70% capacity, rotate to a new filter. During rotation: new URLs go to the new filter; old filter kept read-only for 30 days to catch duplicates of recently-crawled pages. Counting Bloom filter (supports deletions) if you need to remove URLs that have been re-crawled and should be treated as fresh
detect exact duplicates using SHA-256 of page content. Misses near-duplicates — the same article republished on 50 domains with minor wording differences all get indexed, wasting storage and diluting search quality
SimHash fingerprint. Process: tokenize → tf-idf weighted term vector → hash each term with a random 64-bit weight vector → sum → take sign of each bit. Result: 64-bit fingerprint where similar documents have high Hamming similarity. Hamming distance < 3 = near-duplicate
lookup efficiency: comparing a new fingerprint against all 5B stored fingerprints is O(N). Solution: partition fingerprints into bands (groups of bits) and only compare fingerprints in the same band — Locality Sensitive Hashing. Reduces lookup from O(N) to O(1) average case. For 5B fingerprints × 8 bytes = 40 GB total — fits in a distributed Redis cluster for fast lookups
Why the deep dives connect to the scaling problem: "Massive frontier, external bottlenecks, duplicate avoidance, fault tolerance." Each deep dive addresses one constraint.
+-------------------+
| Seed URL Input |
+---------+---------+
|
v
+-------------------------------+
| URL Frontier Queue SQS |
+-------------------------------+
|
dequeue URL msg
|
v
+---------------------------------------------------+
| URL Fetcher Workers |
| - check URL dedup in Metadata DB |
| - read domain robots rules |
| - acquire per-domain lock in Redis |
| - enforce crawl delay and rate limit |
| - resolve DNS and fetch page |
+-------------------+-------------------------------+
| |
disallowed or delayed | fetched HTML
| v
| +------------------------+
| | Raw HTML Blob Storage |
| | S3 |
| +-----------+------------+
| |
| v
| +-------------------------------+
| | Processing Queue |
| | URL id or blob pointer |
| +---------------+---------------+
| |
| v
| +-------------------------------+
| | Text and URL Extraction |
| | Workers |
| | - parse HTML |
| | - extract text |
| | - extract outgoing links |
| | - hash content for dedup |
| +----------+----------+---------+
| | |
| | |
| | v
| | +-------------------+
| | | New URL Discovery |
| | +---------+---------+
| | |
| | check seen URL
| | check depth limit
| | |
| | v
| | +--------------------+
| | | URL Frontier Queue |
| | +--------------------+
| |
| v
| +-------------------------------+
| | Text Blob Storage S3 |
| +-------------------------------+
|
v
+-------------------------------+
| Retry with Backoff |
| SQS visibility timeout + DLQ |
+-------------------------------+
+--------------------+ +----------------------+ +------------------+
| Metadata DB | | Redis | | DNS Cache |
| - URL state | | - per-domain lock | | - domain to IP |
| - crawl depth | | - rate limiting | +------------------+
| - html pointer | +----------------------+
| - text pointer |
| - content hash |
| - robots rules |
| - last crawl time |
+--------------------+
Outside system
+------------------+ +------------------+
| DNS Providers | | External Websites|
+------------------+ +------------------+If you are presenting this in an interview, the simplest way to walk through it is this. URLs enter the frontier queue, fetchers crawl pages politely, raw HTML goes to blob storage, parser workers extract text and links, text goes to storage, and new links go back into the frontier after dedup checks.
The two things that make this feel complete are the control plane pieces. Metadata DB tracks crawl state and dedup, while Redis handles per-domain coordination so you do not overload a site.
Problem
Systematically discovering and fetching web pages. Handle billions of URLs, respect site rate limits, avoid re-crawling duplicates, and recover from failures.
Hard parts: politeness, deduplication at scale, and prioritizing fresh content.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in Web Crawler is not storing pages. It is coordinating a huge number of fetches across the public internet without wasting work or being rude to other sites.
There are four scaling pain points you should call out. First, the crawl frontier gets huge. You keep discovering new URLs, but you need to dedupe them so different workers do not crawl the same page again and again. Second, politeness limits parallelism. You may want massive throughput, but you still need per domain rate limits and robots.txt checks, so scaling is not just adding more workers. Third, the internet is unreliable. DNS lookups, slow servers, dead links, retries, and crawler traps all waste time unless you pipeline the work and track progress carefully. Fourth, the workload is very uneven. Some domains are tiny, some are enormous, and some generate endless near-duplicate pages, so load balancing is messy.
A good interview summary is this. Web Crawler is hard because it combines massive frontier management, external bottlenecks like DNS and website limits, duplicate avoidance, and fault tolerance in one system.
Key points
- Scope it first. Core: crawl the web, extract text, store pages, enable search indexing. Out of scope unless asked: JavaScript rendering, login-gated content, real-time re-crawl, entity extraction.
- URL frontier is the core data structure. Two-tier priority queue. Back queues: one per domain (enforces politeness — one request per domain per N seconds). Front queue: priority-ordered list of domains to crawl next. Never crawl faster than the site allows.
- Always respect robots.txt. Legal requirement in most jurisdictions. Cache per domain (TTL 24h). Check before every fetch. Violation = IP ban. State this proactively in interviews — it signals production awareness.
- Bloom filter for URL dedup. 96K new URL candidates/sec. DB-only dedup = impossible. Bloom filter (9.6 GB for 5B URLs at 1% FPR): O(1) per check. False positive = occasionally skip a valid URL — acceptable. Cassandra exact check only for Bloom positives.
- DNS caching per domain. At 1,929 fetches/sec, uncached DNS at 100ms per lookup = DNS becomes the bottleneck. Cache per domain with TTL (1 hour). Pre-fetch DNS for domains in the near-term frontier queue. DNS miss rate target < 5%.
- Content dedup — SimHash. 30-40% of the web is near-duplicate content. SimHash fingerprint: 64-bit, Hamming distance < 3 = near-duplicate. Partition fingerprints by first K bits for efficient lookup (LSH). Canonical URL from <link rel=canonical> takes priority.
- Failure mode to name. Crawler enters a trap (infinite URL space): per-domain URL count limit (1M max), path depth limit (10 levels), counter-pattern detection. Without traps, one rogue domain can starve the entire crawl budget.
Tradeoffs
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
a single queue of URLs to crawl in BFS order — simple and correct for small scale
two-tier priority frontier. Back queues: one per domain, enforces politeness (one request per domain per N seconds, respects robots.txt Crawl-delay). Front queue: priority-ordered list of (domain, priority) pairs. The scheduler picks the highest-priority domain whose next_allowed_crawl_time ≤ now. This ensures politeness and value-maximization simultaneously
priority = f(PageRank estimate, domain authority, freshness score). High-authority domains (CNN, Wikipedia) get crawled most frequently. Pages within a domain are prioritized by inbound link count. The priority function is configurable — it's the lever that determines what percentage of your crawl budget goes to fresh high-value content vs. long-tail pages
store all crawled URLs in a database, check before each fetch. At 96K new URL candidates/sec, 96K DB queries/sec for dedup alone — impossible
Bloom filter as a fast first-pass. At 5B URLs, 1% false positive rate: 9.6 GB memory, 7 hash functions, O(1) per check. False positive = occasionally skipping a valid URL — acceptable tradeoff. Cassandra exact check only for Bloom filter positives (~1% of checks)
operational concern: Bloom filter fills up. Monitor load factor — alert at 70% capacity, rotate to a new filter. During rotation: new URLs go to the new filter; old filter kept read-only for 30 days to catch duplicates of recently-crawled pages. Counting Bloom filter (supports deletions) if you need to remove URLs that have been re-crawled and should be treated as fresh
detect exact duplicates using SHA-256 of page content. Misses near-duplicates — the same article republished on 50 domains with minor wording differences all get indexed, wasting storage and diluting search quality
SimHash fingerprint. Process: tokenize → tf-idf weighted term vector → hash each term with a random 64-bit weight vector → sum → take sign of each bit. Result: 64-bit fingerprint where similar documents have high Hamming similarity. Hamming distance < 3 = near-duplicate
lookup efficiency: comparing a new fingerprint against all 5B stored fingerprints is O(N). Solution: partition fingerprints into bands (groups of bits) and only compare fingerprints in the same band — Locality Sensitive Hashing. Reduces lookup from O(N) to O(1) average case. For 5B fingerprints × 8 bytes = 40 GB total — fits in a distributed Redis cluster for fast lookups
Why the deep dives connect to the scaling problem: "Massive frontier, external bottlenecks, duplicate avoidance, fault tolerance." Each deep dive addresses one constraint.
Interview script
Whiteboard
+-------------------+
| Seed URL Input |
+---------+---------+
|
v
+-------------------------------+
| URL Frontier Queue SQS |
+-------------------------------+
|
dequeue URL msg
|
v
+---------------------------------------------------+
| URL Fetcher Workers |
| - check URL dedup in Metadata DB |
| - read domain robots rules |
| - acquire per-domain lock in Redis |
| - enforce crawl delay and rate limit |
| - resolve DNS and fetch page |
+-------------------+-------------------------------+
| |
disallowed or delayed | fetched HTML
| v
| +------------------------+
| | Raw HTML Blob Storage |
| | S3 |
| +-----------+------------+
| |
| v
| +-------------------------------+
| | Processing Queue |
| | URL id or blob pointer |
| +---------------+---------------+
| |
| v
| +-------------------------------+
| | Text and URL Extraction |
| | Workers |
| | - parse HTML |
| | - extract text |
| | - extract outgoing links |
| | - hash content for dedup |
| +----------+----------+---------+
| | |
| | |
| | v
| | +-------------------+
| | | New URL Discovery |
| | +---------+---------+
| | |
| | check seen URL
| | check depth limit
| | |
| | v
| | +--------------------+
| | | URL Frontier Queue |
| | +--------------------+
| |
| v
| +-------------------------------+
| | Text Blob Storage S3 |
| +-------------------------------+
|
v
+-------------------------------+
| Retry with Backoff |
| SQS visibility timeout + DLQ |
+-------------------------------+
+--------------------+ +----------------------+ +------------------+
| Metadata DB | | Redis | | DNS Cache |
| - URL state | | - per-domain lock | | - domain to IP |
| - crawl depth | | - rate limiting | +------------------+
| - html pointer | +----------------------+
| - text pointer |
| - content hash |
| - robots rules |
| - last crawl time |
+--------------------+
Outside system
+------------------+ +------------------+
| DNS Providers | | External Websites|
+------------------+ +------------------+If you are presenting this in an interview, the simplest way to walk through it is this. URLs enter the frontier queue, fetchers crawl pages politely, raw HTML goes to blob storage, parser workers extract text and links, text goes to storage, and new links go back into the frontier after dedup checks.
The two things that make this feel complete are the control plane pieces. Metadata DB tracks crawl state and dedup, while Redis handles per-domain coordination so you do not overload a site.
ad_id
Counting ad clicks accurately at massive scale for billing advertisers.
The core challenge: exact counting at this scale is expensive. The system must balance real-time approximation vs exact billing accuracy.
The hard part in Ad Click Aggregator is that it looks like a simple counter system, but it is really a high write analytics pipeline with correctness requirements.
There are three main scaling pain points. First, the write path is heavy. Every click is an event, so you cannot do raw database writes and then run GROUP BY queries on demand. You need to buffer and pre aggregate the data. Second, freshness matters. Advertisers want near real time metrics, so pure batch processing makes data too stale, which pushes you toward streaming aggregation. Third, correctness matters a lot because clicks map to money. You cannot lose events, and you also do not want to double count duplicate clicks.
A fourth issue to call out is skew. Most ads are quiet, but one viral ad can create a hot shard if all clicks for that ad land on the same partition. That means the system is hard not because 10k clicks per second is huge by itself, but because you need high write throughput, low latency analytics, and accurate counting all at once.
A good interview summary is this. Ad Click Aggregator is hard because it combines write heavy ingestion, near real time aggregation, idempotency, and hot key traffic in one pipeline.
- Lambda Architecture. Real-time path for fast approximate aggregates. Batch path for exact billing reconciliation. Name both paths explicitly.
- Click dedup is critical. Each click has a UUID. Bloom filter at ingestion for fast dedup. Prevents inflated advertiser bills.
- Flink watermarking. Allow up to 1 minute of lateness. Bounds output latency.
- ClickHouse for OLAP. Columnar store optimized for aggregation queries.
- Batch for billing. Billing must use exact counts. Daily Spark job reconciles.
The business requirement creates the architectural constraint: real-time approximate metrics for dashboards (advertisers want to see campaign performance now) AND exact counts for billing (advertisers dispute invoices with exact numbers). No single pipeline satisfies both
stream only (approximate)
Lambda Architecture explicitly. Speed layer (Flink): processes Kafka events in near-real-time, aggregates with 100ms batch windows, writes to ClickHouse. Results are fast but approximate (Flink checkpointing can reprocess, but windows have an allowed lateness boundary beyond which events are dropped). Batch layer (Spark): reads raw events from the event lake (S3-compatible, 90-day retention), runs daily exact aggregation. Batch results are exact, with 24-hour latency. Serving layer (ClickHouse): stores both approximate (updated every 30s by Flink) and exact (updated daily by Spark) counts. Advertisers see approximate for real-time view, exact for invoice
architectural point: the batch layer is not a fallback — it's a first-class part of the design that serves a different SLA requirement
Click fraud via duplicate clicks directly translates to advertiser overbilling. Dedup must be accurate
check DB for duplicates
UUID per click event, Bloom filter at ingestion. The Bloom filter: expected 100M unique events/day, 0.01% false positive rate → 300 MB memory, acceptably small. On each click event: check Bloom filter. If not present: pass through, add to Bloom filter. If present: probable duplicate, drop the event. Bloom filter false positives (0.01%) = 10K legitimate clicks dropped per day out of 100M — acceptable for real-time dashboard. For billing-critical accuracy: the batch path uses exact dedup. The raw event log in S3 is the source of truth. Spark job: GROUP BY (click_id, UUID) and count distinct — exact dedup. Any click that appears in the raw log but not in the Bloom-filtered stream is captured in the batch reconciliation
the Bloom filter for a 24-hour window is reset daily. Old Bloom filters are discarded, new ones start fresh. UUID expiry aligns with the billing period
increase the total number of Kafka partitions cluster-wide
one viral ad campaign can generate 100× normal click volume. In a Kafka cluster partitioned by ad_id, this creates a hot partition
detection: monitor consumer lag per (topic, partition). When a partition exceeds a lag threshold, detect the hot ad_id causing the spike. Use a compound partition key (ad_id + random_suffix_0_to_N) to spread the load across N partitions. The Flink job handles the merge: KEY BY ad_id across multiple partitions gives correct aggregation regardless of how many partitions the ad's events are spread across. Weak answer: increase partition count. Strong answer: dynamic repartitioning. Monitor partition consumer lag per (topic, partition). When a partition exceeds a lag threshold: detect the hot ad_id causing the spike. Create additional Kafka partitions for that ad_id using a compound key (ad_id + random_suffix). The Flink job handles this: multiple partitions for the same ad_id, Flink aggregates across all partitions in a keyed stream (KEY BY ad_id → Flink handles the merge). The output is correct regardless of how many partitions the ad's events are spread across. At the ClickHouse write side: batching (Flink emits aggregated counts rather than raw events) reduces ClickHouse write amplification. For ClickHouse: ad_id is the partition key for the aggregated table — hot ad_id creates a hot ClickHouse partition. Fix: ReplicatedMergeTree with multiple replicas for hot ad_ids (ClickHouse routes reads to replicas)
Why the deep dives connect to the scaling problem: "Write-heavy analytics pipeline with correctness requirements." Each deep dive addresses one constraint.
+----------------------+
| Ad Placement Svc |
| returns ad, target |
| impressionId, sig |
+----------+-----------+
|
v
+---------+ click ad +--------------------+
| User | ---------------------> | Click Endpoint |
| Browser | | /click |
+----+----+ +-----+----------+---+
^ | |
| 302 redirect to advertiser | |
| | |
| | v
| | +-------------+
| | | Signature |
| | | Verification|
| | +------+------+
| | |
| | v
| | +-------------+
| | | Redis Cache |
| | | dedup by |
| | | impressionId|
| | +-------+-----+
| | |
| | duplicate?|
| | yes -> drop
| | |
| v no
| +-----------------------+
| | Kafka or Kinesis |
| | durable click stream |
| +-----+------------+----+
| | |
| | +------------------+
| | |
| v v
| +-------------------+ +-------------------+
| | Flink stream proc | | S3 data lake |
| | window by minute | | raw click archive |
| | aggregate clicks | +---------+---------+
| +---------+---------+ |
| | |
| v v
| +-------------------+ +-------------------+
| | OLAP analytics DB | <--------| Batch reconcile |
| | ClickHouse, | | Spark daily or |
| | BigQuery, etc | | hourly recompute |
| +---------+---------+ +-------------------+
| |
| v
| +-------------------+
+-------------------------| Advertiser Query |
| dashboard or API |
+-------------------+The mental model is two paths. The serving path handles the user click and redirect fast, and the analytics path turns raw clicks into queryable per minute metrics.
If you were drawing this in an interview, I would keep the main story to five boxes first. User, Click Endpoint, Stream, Stream Processor, OLAP DB. Then add Redis for dedup and S3 plus batch reconcile only if the interviewer asks about idempotency or correctness.
Problem
Counting ad clicks accurately at massive scale for billing advertisers.
The core challenge: exact counting at this scale is expensive. The system must balance real-time approximation vs exact billing accuracy.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in Ad Click Aggregator is that it looks like a simple counter system, but it is really a high write analytics pipeline with correctness requirements.
There are three main scaling pain points. First, the write path is heavy. Every click is an event, so you cannot do raw database writes and then run GROUP BY queries on demand. You need to buffer and pre aggregate the data. Second, freshness matters. Advertisers want near real time metrics, so pure batch processing makes data too stale, which pushes you toward streaming aggregation. Third, correctness matters a lot because clicks map to money. You cannot lose events, and you also do not want to double count duplicate clicks.
A fourth issue to call out is skew. Most ads are quiet, but one viral ad can create a hot shard if all clicks for that ad land on the same partition. That means the system is hard not because 10k clicks per second is huge by itself, but because you need high write throughput, low latency analytics, and accurate counting all at once.
A good interview summary is this. Ad Click Aggregator is hard because it combines write heavy ingestion, near real time aggregation, idempotency, and hot key traffic in one pipeline.
Key points
- Lambda Architecture. Real-time path for fast approximate aggregates. Batch path for exact billing reconciliation. Name both paths explicitly.
- Click dedup is critical. Each click has a UUID. Bloom filter at ingestion for fast dedup. Prevents inflated advertiser bills.
- Flink watermarking. Allow up to 1 minute of lateness. Bounds output latency.
- ClickHouse for OLAP. Columnar store optimized for aggregation queries.
- Batch for billing. Billing must use exact counts. Daily Spark job reconciles.
Tradeoffs
Deep dives
The business requirement creates the architectural constraint: real-time approximate metrics for dashboards (advertisers want to see campaign performance now) AND exact counts for billing (advertisers dispute invoices with exact numbers). No single pipeline satisfies both
stream only (approximate)
Lambda Architecture explicitly. Speed layer (Flink): processes Kafka events in near-real-time, aggregates with 100ms batch windows, writes to ClickHouse. Results are fast but approximate (Flink checkpointing can reprocess, but windows have an allowed lateness boundary beyond which events are dropped). Batch layer (Spark): reads raw events from the event lake (S3-compatible, 90-day retention), runs daily exact aggregation. Batch results are exact, with 24-hour latency. Serving layer (ClickHouse): stores both approximate (updated every 30s by Flink) and exact (updated daily by Spark) counts. Advertisers see approximate for real-time view, exact for invoice
architectural point: the batch layer is not a fallback — it's a first-class part of the design that serves a different SLA requirement
Click fraud via duplicate clicks directly translates to advertiser overbilling. Dedup must be accurate
check DB for duplicates
UUID per click event, Bloom filter at ingestion. The Bloom filter: expected 100M unique events/day, 0.01% false positive rate → 300 MB memory, acceptably small. On each click event: check Bloom filter. If not present: pass through, add to Bloom filter. If present: probable duplicate, drop the event. Bloom filter false positives (0.01%) = 10K legitimate clicks dropped per day out of 100M — acceptable for real-time dashboard. For billing-critical accuracy: the batch path uses exact dedup. The raw event log in S3 is the source of truth. Spark job: GROUP BY (click_id, UUID) and count distinct — exact dedup. Any click that appears in the raw log but not in the Bloom-filtered stream is captured in the batch reconciliation
the Bloom filter for a 24-hour window is reset daily. Old Bloom filters are discarded, new ones start fresh. UUID expiry aligns with the billing period
increase the total number of Kafka partitions cluster-wide
one viral ad campaign can generate 100× normal click volume. In a Kafka cluster partitioned by ad_id, this creates a hot partition
detection: monitor consumer lag per (topic, partition). When a partition exceeds a lag threshold, detect the hot ad_id causing the spike. Use a compound partition key (ad_id + random_suffix_0_to_N) to spread the load across N partitions. The Flink job handles the merge: KEY BY ad_id across multiple partitions gives correct aggregation regardless of how many partitions the ad's events are spread across. Weak answer: increase partition count. Strong answer: dynamic repartitioning. Monitor partition consumer lag per (topic, partition). When a partition exceeds a lag threshold: detect the hot ad_id causing the spike. Create additional Kafka partitions for that ad_id using a compound key (ad_id + random_suffix). The Flink job handles this: multiple partitions for the same ad_id, Flink aggregates across all partitions in a keyed stream (KEY BY ad_id → Flink handles the merge). The output is correct regardless of how many partitions the ad's events are spread across. At the ClickHouse write side: batching (Flink emits aggregated counts rather than raw events) reduces ClickHouse write amplification. For ClickHouse: ad_id is the partition key for the aggregated table — hot ad_id creates a hot ClickHouse partition. Fix: ReplicatedMergeTree with multiple replicas for hot ad_ids (ClickHouse routes reads to replicas)
Why the deep dives connect to the scaling problem: "Write-heavy analytics pipeline with correctness requirements." Each deep dive addresses one constraint.
Interview script
Whiteboard
+----------------------+
| Ad Placement Svc |
| returns ad, target |
| impressionId, sig |
+----------+-----------+
|
v
+---------+ click ad +--------------------+
| User | ---------------------> | Click Endpoint |
| Browser | | /click |
+----+----+ +-----+----------+---+
^ | |
| 302 redirect to advertiser | |
| | |
| | v
| | +-------------+
| | | Signature |
| | | Verification|
| | +------+------+
| | |
| | v
| | +-------------+
| | | Redis Cache |
| | | dedup by |
| | | impressionId|
| | +-------+-----+
| | |
| | duplicate?|
| | yes -> drop
| | |
| v no
| +-----------------------+
| | Kafka or Kinesis |
| | durable click stream |
| +-----+------------+----+
| | |
| | +------------------+
| | |
| v v
| +-------------------+ +-------------------+
| | Flink stream proc | | S3 data lake |
| | window by minute | | raw click archive |
| | aggregate clicks | +---------+---------+
| +---------+---------+ |
| | |
| v v
| +-------------------+ +-------------------+
| | OLAP analytics DB | <--------| Batch reconcile |
| | ClickHouse, | | Spark daily or |
| | BigQuery, etc | | hourly recompute |
| +---------+---------+ +-------------------+
| |
| v
| +-------------------+
+-------------------------| Advertiser Query |
| dashboard or API |
+-------------------+The mental model is two paths. The serving path handles the user click and redirect fast, and the analytics path turns raw clicks into queryable per minute metrics.
If you were drawing this in an interview, I would keep the main story to five boxes first. User, Click Endpoint, Stream, Stream Processor, OLAP DB. Then add Redis for dedup and S3 plus batch reconcile only if the interviewer asks about idempotency or correctness.
no double-schedule
Running scheduled and dependency-based jobs reliably.
Hard parts: preventing double-scheduling, enforcing DAG dependencies, and ensuring jobs complete even when workers crash.
The hard part in Job Scheduler is time. You are not just storing jobs. You have to find the right jobs at the right moment, execute them close to their scheduled time, and still keep the system durable when workers crash.
There are three main scaling pain points. First, time based lookup gets expensive. If you store recurring schedules like cron expressions, you cannot scan every job every second to see what is due. That is why you usually separate the job definition from execution instances and index executions by time. Second, precision and throughput fight each other. At 10k jobs per second, polling the database very frequently creates huge read load, but polling less often makes jobs late. A common fix is a two phase design where the database holds durable schedule state and a queue handles near term delivery. Third, retries and failures create duplicate work. If a worker dies mid job, the system must retry, which means you need at least once delivery and idempotent tasks so running a job twice does not break things.
A good interview summary is this. Job Scheduler is hard to scale because it combines time based querying, high throughput dispatch, and failure handling in one system. The system has to be both precise like a clock and resilient like a queue.
- Scope it first. Core: define jobs with schedules (cron or interval), define DAG dependencies, execute jobs reliably, retry on failure, monitor status. Out of scope unless asked: live streaming jobs, sub-second scheduling, multi-tenant isolation.
- Single leader — no double-dispatch. Leader election via etcd or ZooKeeper. Only the leader polls the DB for due jobs and enqueues them. Two schedulers running simultaneously = same job enqueued twice = duplicate execution. Single leader is the correct default.
- Pull model — workers poll SQS. Workers signal availability by pulling from SQS. SQS handles capacity naturally: idle workers drain queue, busy workers don't pull. Scheduler needs no knowledge of worker count or health. Scale workers independently.
- Heartbeat requeue for fault tolerance. Worker sends heartbeat every 30s. Scheduler monitors: if heartbeat stops for 60s, mark job as timed out and re-enqueue. At-least-once execution — jobs must be idempotent (check "already ran" at start).
- DAG validation at definition time. Topological sort on DAG definition. Cycle detected → reject with error. Never store an invalid DAG. Cycle detection at runtime is too late — jobs would loop forever without a natural termination condition.
- Dead letter queue for permanent failures. Max retries (e.g., 3) with exponential backoff. After max retries: move to DLQ, alert on-call. Never silently discard failed jobs. DLQ entries visible in UI for manual re-trigger after root cause fix.
- Failure mode to name. Leader crashes mid-dispatch: SQS deduplication ID prevents duplicate enqueue (idempotent). New leader is elected in <1s (etcd watch-based). Jobs in-flight continue executing under their existing workers — no interruption.
run multiple scheduler instances in parallel for redundancy
two scheduler instances running simultaneously would both scan the same job table and enqueue the same jobs twice. Workers execute jobs twice → idempotency violations, corrupted state, duplicate emails, double payments. Weak answer: use a DB lock. Strong answer: ZooKeeper or etcd leader election. etcd approach: all scheduler instances compete to create an ephemeral key /scheduler/leader with their instance ID. The TTL is 15 seconds (heartbeat interval). Only one instance can create the key — that instance is the leader. Other instances watch the key and wait. If the leader crashes: key expires in 15 seconds, election re-runs
DB-based election (UPDATE schedulers SET is_leader=1, heartbeat_at=now() WHERE id=? AND is_leader=0) works but has 30-60 second failover (depends on heartbeat check frequency) and adds polling load to PG. etcd/ZooKeeper: event-driven (watch-based), <1 second failover, designed for coordination. The choice is operational: if you already have etcd in your stack (Kubernetes uses it), use it. If not, DB-based is acceptable
Airflow-style DAGs: task B can only run after task A succeeds. At 10M executions/day with complex DAGs (some with 50+ tasks), the scheduler must efficiently find tasks that are ready to run
scan all tasks periodically
event-driven dependency resolution. When a task completes: publish a TASK_COMPLETED event. The scheduler consumes this event, checks if all dependencies for downstream tasks are now satisfied, and enqueues ready tasks. Dependency check: SELECT count(*) FROM task_instances WHERE dag_run_id=? AND task_id IN (upstream_tasks) AND status != 'SUCCESS'. If count = 0: all upstreams succeeded, enqueue the task
this check is a hotspot under high fan-out DAGs (one task → 100 downstream tasks). Batch the dependency check: on TASK_COMPLETED, add the dag_run_id to a Redis set. A low-frequency background scanner processes the set, checks all downstream tasks for that dag_run, enqueues ready ones. Reduces per-event DB queries from O(downstream_tasks) to O(1) per event. At-least-once delivery: if the dependency check fails (DB unavailable), the task remains in PENDING state. The periodic scanner catches it on the next cycle
mark a job as failed only when the worker explicitly reports failure
a worker executing a job may crash mid-execution. The job must be retried
at-least-once design: worker sends heartbeat every 30 seconds (UPDATE task_instances SET last_heartbeat=now() WHERE id=? AND status='RUNNING'). Scheduler scans for stale heartbeats: SELECT id FROM task_instances WHERE status='RUNNING' AND last_heartbeat < now() - INTERVAL '60s'. Re-enqueues timed-out tasks. This gives at-least-once execution — the job may run twice if the worker crashes and recovers but the heartbeat was temporarily delayed. Exactly-once requires idempotent jobs: a job that can be safely run twice must produce the same result (send-email with deduplication ID, db-insert with upsert, file-generation with atomic rename). Staff+ design principle: the scheduler guarantees at-least-once; job authors are responsible for idempotency. This is an explicit contract documented in the platform's API. For non-idempotent jobs: add an explicit "already-ran" check at job start (SELECT 1 FROM job_executions WHERE job_id=? AND execution_date=? AND status='SUCCEEDED'). Dead letter queue for jobs that fail beyond max_retries — never silently discard
Why the deep dives connect to the scaling problem: "Time-based querying, high-throughput dispatch, failure handling." Each deep dive addresses one constraint.
+-------------------+
| User |
+---------+---------+
|
POST /jobs
GET /jobs
|
v
+------------+------------+
| API Service |
+------------+------------+
|
+-----------------+-----------------+
| |
v v
+--------+--------+ +--------+---------+
| Jobs Table | | Executions Table |
| job definition | | run instances |
+--------+--------+ +--------+---------+
| |
| GSI on user_id + time
| |
| v
| +---------+---------+
| | Status Query Path |
| +-------------------+
|
| every 5 min scans next ~5 min
v
+--------+--------+
| Scheduler Cron |
| / Dispatcher |
+--------+--------+
|
| enqueue with delay
v
+--------+---------+
| Delayed Queue |
| SQS / Redis |
+--------+---------+
|
| messages become visible near run time
v
+----------+----------+------------+
| | |
v v v
+----+-----+ +----+-----+ +---+------+
| Worker A | | Worker B | | Worker N |
+----+-----+ +----+-----+ +---+------+
| | |
+----------+----------+------------+
|
| fetch job details
v
+--------+--------+
| Jobs Table |
+--------+--------+
|
| execute task
v
+--------+--------+
| Task Handler(s) |
| email, webhook, |
| cleanup, etc. |
+--------+--------+
|
+---------+----------+
| |
v v
+-------+-------+ +-------+--------+
| success | | failure |
| mark complete | | retry w backoff|
+-------+-------+ +-------+--------+
| |
+---------+----------+
|
v
+--------+---------+
| Executions Table |
| status updates |
+------------------+Draw the two tables first — that is the data model. Then draw the Scheduler scanning and enqueueing. Then the worker pool. Then the success/failure split at the bottom. Save the Status Query Path and GSI for if the interviewer asks about read patterns.
Problem
Running scheduled and dependency-based jobs reliably.
Hard parts: preventing double-scheduling, enforcing DAG dependencies, and ensuring jobs complete even when workers crash.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in Job Scheduler is time. You are not just storing jobs. You have to find the right jobs at the right moment, execute them close to their scheduled time, and still keep the system durable when workers crash.
There are three main scaling pain points. First, time based lookup gets expensive. If you store recurring schedules like cron expressions, you cannot scan every job every second to see what is due. That is why you usually separate the job definition from execution instances and index executions by time. Second, precision and throughput fight each other. At 10k jobs per second, polling the database very frequently creates huge read load, but polling less often makes jobs late. A common fix is a two phase design where the database holds durable schedule state and a queue handles near term delivery. Third, retries and failures create duplicate work. If a worker dies mid job, the system must retry, which means you need at least once delivery and idempotent tasks so running a job twice does not break things.
A good interview summary is this. Job Scheduler is hard to scale because it combines time based querying, high throughput dispatch, and failure handling in one system. The system has to be both precise like a clock and resilient like a queue.
Key points
- Scope it first. Core: define jobs with schedules (cron or interval), define DAG dependencies, execute jobs reliably, retry on failure, monitor status. Out of scope unless asked: live streaming jobs, sub-second scheduling, multi-tenant isolation.
- Single leader — no double-dispatch. Leader election via etcd or ZooKeeper. Only the leader polls the DB for due jobs and enqueues them. Two schedulers running simultaneously = same job enqueued twice = duplicate execution. Single leader is the correct default.
- Pull model — workers poll SQS. Workers signal availability by pulling from SQS. SQS handles capacity naturally: idle workers drain queue, busy workers don't pull. Scheduler needs no knowledge of worker count or health. Scale workers independently.
- Heartbeat requeue for fault tolerance. Worker sends heartbeat every 30s. Scheduler monitors: if heartbeat stops for 60s, mark job as timed out and re-enqueue. At-least-once execution — jobs must be idempotent (check "already ran" at start).
- DAG validation at definition time. Topological sort on DAG definition. Cycle detected → reject with error. Never store an invalid DAG. Cycle detection at runtime is too late — jobs would loop forever without a natural termination condition.
- Dead letter queue for permanent failures. Max retries (e.g., 3) with exponential backoff. After max retries: move to DLQ, alert on-call. Never silently discard failed jobs. DLQ entries visible in UI for manual re-trigger after root cause fix.
- Failure mode to name. Leader crashes mid-dispatch: SQS deduplication ID prevents duplicate enqueue (idempotent). New leader is elected in <1s (etcd watch-based). Jobs in-flight continue executing under their existing workers — no interruption.
Tradeoffs
Deep dives
run multiple scheduler instances in parallel for redundancy
two scheduler instances running simultaneously would both scan the same job table and enqueue the same jobs twice. Workers execute jobs twice → idempotency violations, corrupted state, duplicate emails, double payments. Weak answer: use a DB lock. Strong answer: ZooKeeper or etcd leader election. etcd approach: all scheduler instances compete to create an ephemeral key /scheduler/leader with their instance ID. The TTL is 15 seconds (heartbeat interval). Only one instance can create the key — that instance is the leader. Other instances watch the key and wait. If the leader crashes: key expires in 15 seconds, election re-runs
DB-based election (UPDATE schedulers SET is_leader=1, heartbeat_at=now() WHERE id=? AND is_leader=0) works but has 30-60 second failover (depends on heartbeat check frequency) and adds polling load to PG. etcd/ZooKeeper: event-driven (watch-based), <1 second failover, designed for coordination. The choice is operational: if you already have etcd in your stack (Kubernetes uses it), use it. If not, DB-based is acceptable
Airflow-style DAGs: task B can only run after task A succeeds. At 10M executions/day with complex DAGs (some with 50+ tasks), the scheduler must efficiently find tasks that are ready to run
scan all tasks periodically
event-driven dependency resolution. When a task completes: publish a TASK_COMPLETED event. The scheduler consumes this event, checks if all dependencies for downstream tasks are now satisfied, and enqueues ready tasks. Dependency check: SELECT count(*) FROM task_instances WHERE dag_run_id=? AND task_id IN (upstream_tasks) AND status != 'SUCCESS'. If count = 0: all upstreams succeeded, enqueue the task
this check is a hotspot under high fan-out DAGs (one task → 100 downstream tasks). Batch the dependency check: on TASK_COMPLETED, add the dag_run_id to a Redis set. A low-frequency background scanner processes the set, checks all downstream tasks for that dag_run, enqueues ready ones. Reduces per-event DB queries from O(downstream_tasks) to O(1) per event. At-least-once delivery: if the dependency check fails (DB unavailable), the task remains in PENDING state. The periodic scanner catches it on the next cycle
mark a job as failed only when the worker explicitly reports failure
a worker executing a job may crash mid-execution. The job must be retried
at-least-once design: worker sends heartbeat every 30 seconds (UPDATE task_instances SET last_heartbeat=now() WHERE id=? AND status='RUNNING'). Scheduler scans for stale heartbeats: SELECT id FROM task_instances WHERE status='RUNNING' AND last_heartbeat < now() - INTERVAL '60s'. Re-enqueues timed-out tasks. This gives at-least-once execution — the job may run twice if the worker crashes and recovers but the heartbeat was temporarily delayed. Exactly-once requires idempotent jobs: a job that can be safely run twice must produce the same result (send-email with deduplication ID, db-insert with upsert, file-generation with atomic rename). Staff+ design principle: the scheduler guarantees at-least-once; job authors are responsible for idempotency. This is an explicit contract documented in the platform's API. For non-idempotent jobs: add an explicit "already-ran" check at job start (SELECT 1 FROM job_executions WHERE job_id=? AND execution_date=? AND status='SUCCEEDED'). Dead letter queue for jobs that fail beyond max_retries — never silently discard
Why the deep dives connect to the scaling problem: "Time-based querying, high-throughput dispatch, failure handling." Each deep dive addresses one constraint.
Interview script
Whiteboard
+-------------------+
| User |
+---------+---------+
|
POST /jobs
GET /jobs
|
v
+------------+------------+
| API Service |
+------------+------------+
|
+-----------------+-----------------+
| |
v v
+--------+--------+ +--------+---------+
| Jobs Table | | Executions Table |
| job definition | | run instances |
+--------+--------+ +--------+---------+
| |
| GSI on user_id + time
| |
| v
| +---------+---------+
| | Status Query Path |
| +-------------------+
|
| every 5 min scans next ~5 min
v
+--------+--------+
| Scheduler Cron |
| / Dispatcher |
+--------+--------+
|
| enqueue with delay
v
+--------+---------+
| Delayed Queue |
| SQS / Redis |
+--------+---------+
|
| messages become visible near run time
v
+----------+----------+------------+
| | |
v v v
+----+-----+ +----+-----+ +---+------+
| Worker A | | Worker B | | Worker N |
+----+-----+ +----+-----+ +---+------+
| | |
+----------+----------+------------+
|
| fetch job details
v
+--------+--------+
| Jobs Table |
+--------+--------+
|
| execute task
v
+--------+--------+
| Task Handler(s) |
| email, webhook, |
| cleanup, etc. |
+--------+--------+
|
+---------+----------+
| |
v v
+-------+-------+ +-------+--------+
| success | | failure |
| mark complete | | retry w backoff|
+-------+-------+ +-------+--------+
| |
+---------+----------+
|
v
+--------+---------+
| Executions Table |
| status updates |
+------------------+Draw the two tables first — that is the data model. Then draw the Scheduler scanning and enqueueing. Then the worker pool. Then the success/failure split at the bottom. Save the Status Query Path and GSI for if the interviewer asks about read patterns.
outbox row (same tx)
with retries
Processing payments reliably without double-charging.
Hard parts: idempotency (retries must not double-charge), atomic webhook delivery (outbox pattern), and double-entry accounting.
The hard part in a Payment System is that every request involves real money, so you need both scale and correctness at the same time. A slow feed can be annoying. A duplicated or lost payment is a business disaster.
There are three big scaling pain points. First, the write path is safety critical. At 10k plus TPS, you are creating and updating payment records fast, but you also need idempotency so retries do not double charge a customer. Second, the workflow is asynchronous because external payment networks can timeout or respond later, so your system must track uncertain states and reconcile later instead of assuming success or failure immediately. Third, durability and auditability matter much more than in a normal app. You cannot just keep the latest row state. You need a full history of what happened so you can recover, reconcile, and answer disputes.
A good interview summary is this. Payment Systems are hard to scale because they combine high write throughput, strict financial correctness, and unreliable external dependencies. You are not just processing requests quickly. You are making sure money movement is never lost, duplicated, or misreported.
- Scope it first. Core: accept payments, charge PSP, deliver webhooks to merchants, maintain ledger. Out of scope unless asked: subscriptions, marketplace split payments, refunds (unless asked), tax calculation.
- Idempotency key — store BEFORE PSP call. Client sends UUID. Server stores key as PENDING before calling PSP. On retry: find PENDING → query PSP by idempotency key → update to COMPLETED. If stored after PSP call: crash between them → retry creates duplicate charge.
- Outbox in the same ACID transaction. INSERT INTO payments + INSERT INTO outbox_events in one transaction. Either both commit or neither does. Async worker delivers the webhook. Eliminates lost webhooks without distributed transactions.
- Double-entry accounting — immutable events. Every payment = debit buyer + credit merchant in one transaction. Balance = sum(events). Never UPDATE a balance field. Invariant: sum(all debits + credits) = 0. Auditable, replayable, legally required.
- Shard by merchant_id. All of a merchant's payments on one shard → single-shard ACID for all their transactions, no 2PC needed. Marketplace split payments (cross-merchant) use saga pattern with compensating transactions.
- Reconciliation job. Nightly batch: compare internal ledger against PSP settlement report. Flag any discrepancy for manual review. Financial systems drift — reconciliation catches what the application logic missed.
- Failure mode to name. PSP call times out — did the charge happen? Never assume timeout = failure. Query PSP by idempotency key for status. If PSP confirms: mark COMPLETED. If not found: retry. If ambiguous: hold payment in PENDING and alert ops.
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
check if the payment already exists before processing. At 10K TPS with network timeouts, you can't reliably distinguish "never received" from "received and failed" without an idempotency key
client generates UUID before sending any payment request. Server stores the key with status=PENDING before calling the PSP. On retry: find PENDING key → query PSP by the same idempotency key → update to COMPLETED or FAILED
sequence detail: storing the key AFTER the PSP call is a critical mistake. If the process crashes between PSP success and DB write, the retry has no key to find — it creates a new charge. Store the key BEFORE the PSP call unconditionally. On retry: find PENDING key, call PSP with same idempotency key (PSP deduplicates on its end too), update status. The entire chain is idempotent end-to-end. This sequence is the single most important correctness detail in payment system design
after recording the payment in the DB, make an HTTP call to deliver the webhook. If the process crashes between DB commit and HTTP call, the webhook is lost. The merchant never learns about the payment
outbox pattern. In the same ACID transaction that records the payment: INSERT INTO outbox (event_type, payload, status). Either both the payment and the outbox entry commit, or neither does. An async worker polls the outbox for PENDING entries and delivers them with retries and exponential backoff
idempotency for webhooks: the webhook payload must include an idempotency key. The merchant's endpoint may receive the same webhook multiple times (delivery retry after a timeout). If the merchant's system isn't idempotent, a payment can be processed twice on their end. Document this contract explicitly: webhook delivery is at-least-once, merchant endpoints must be idempotent
use a distributed transaction (2PC) across shards to atomically debit the buyer and credit the seller. 2PC is slow, blocks resources during the coordinator phase, and is prone to blocking on coordinator failure
saga pattern. A saga is a sequence of local transactions, each with a compensating transaction on failure. Payment saga: (1) RESERVE buyer funds (local ACID transaction), (2) CREDIT seller, (3) CAPTURE buyer reservation. On failure at step 2: run compensating transaction for step 1 (release reservation)
durability: the saga state machine must be stored durably. If the saga orchestrator crashes mid-saga, it must resume from the last committed step on restart — not restart from the beginning (which would double-charge). Store saga state in PostgreSQL with the current step and status. On restart: read uncommitted sagas, resume from last committed step. Sagas are eventually consistent — the DB may briefly be in an intermediate state, but all failures are handled gracefully by the compensating transactions
Why the deep dives connect to the scaling problem: "High write throughput, strict financial correctness, unreliable external dependencies." Each deep dive addresses one constraint.
+--------------------+
| Merchant Backend |
| uses API keys |
+---------+----------+
|
| HTTPS
v
+-------------+ +-------+--------+
| Customer | | API Gateway |
| Browser | | auth, routing, |
| checkout UI | | rate limiting |
+------+------+ +---+---------+---+
| | |
| card entry via | |
| hosted iframe / SDK | |
v | |
+------+-----------------------+ |
| Secure Payment SDK / iFrame | |
| card data goes to processor | |
+--------------+---------------+ |
| |
| encrypted card data |
| |
v v
+--------+---------+ +--------+---------+
| Transaction | | PaymentIntent |
| Service | | Service |
| creates charge | | create/read |
| records | | payment intent |
+----+--------+----+ +----+--------+----+
| | | |
| +----------------+ |
| read/write |
v v
+---------------------------------------------+
| Operational Database |
| merchants, payment_intents, transactions, |
| attempts, statuses |
+-------------------+-------------------------+
|
| CDC from DB log
v
+--------+---------+
| Kafka / Event |
| Stream |
| immutable events |
+---+----+----+----+
| | |
| | |
| | +------------------+
| | |
| v v
| +---------+ +-------------+
| | Audit | | Webhook |
| | Service | | Service |
| | history | | notify |
| +----+----+ | merchants |
| | +------+------+
| | |
| v | HTTPS POST
| +---------+ v
| | Cold | +-------------+
| | Storage | | Merchant |
| | S3 etc | | Webhook URL |
| +---------+ +-------------+
|
v
+-------+--------+
| Reconciliation |
| Service |
| resolves |
| timeouts |
+-------+--------+
|
| query status / batch files
v
.---------------------------------------.
| External Payment Networks and Banks |
| Visa, Mastercard, issuing banks |
'---------------------------------------'The mental model is simple. PaymentIntent Service manages the customer payment lifecycle, Transaction Service talks to the outside payment world, and the database plus CDC plus Kafka gives you a durable history so you do not lose money movement events.
If you are drawing this in an interview, you can start with just five boxes. Merchant, API Gateway, PaymentIntent Service, Transaction Service, Database. Then add CDC, Kafka, Reconciliation, and Webhooks only if the interviewer pushes on durability or async safety.
Problem
Processing payments reliably without double-charging.
Hard parts: idempotency (retries must not double-charge), atomic webhook delivery (outbox pattern), and double-entry accounting.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hard part in a Payment System is that every request involves real money, so you need both scale and correctness at the same time. A slow feed can be annoying. A duplicated or lost payment is a business disaster.
There are three big scaling pain points. First, the write path is safety critical. At 10k plus TPS, you are creating and updating payment records fast, but you also need idempotency so retries do not double charge a customer. Second, the workflow is asynchronous because external payment networks can timeout or respond later, so your system must track uncertain states and reconcile later instead of assuming success or failure immediately. Third, durability and auditability matter much more than in a normal app. You cannot just keep the latest row state. You need a full history of what happened so you can recover, reconcile, and answer disputes.
A good interview summary is this. Payment Systems are hard to scale because they combine high write throughput, strict financial correctness, and unreliable external dependencies. You are not just processing requests quickly. You are making sure money movement is never lost, duplicated, or misreported.
Key points
- Scope it first. Core: accept payments, charge PSP, deliver webhooks to merchants, maintain ledger. Out of scope unless asked: subscriptions, marketplace split payments, refunds (unless asked), tax calculation.
- Idempotency key — store BEFORE PSP call. Client sends UUID. Server stores key as PENDING before calling PSP. On retry: find PENDING → query PSP by idempotency key → update to COMPLETED. If stored after PSP call: crash between them → retry creates duplicate charge.
- Outbox in the same ACID transaction. INSERT INTO payments + INSERT INTO outbox_events in one transaction. Either both commit or neither does. Async worker delivers the webhook. Eliminates lost webhooks without distributed transactions.
- Double-entry accounting — immutable events. Every payment = debit buyer + credit merchant in one transaction. Balance = sum(events). Never UPDATE a balance field. Invariant: sum(all debits + credits) = 0. Auditable, replayable, legally required.
- Shard by merchant_id. All of a merchant's payments on one shard → single-shard ACID for all their transactions, no 2PC needed. Marketplace split payments (cross-merchant) use saga pattern with compensating transactions.
- Reconciliation job. Nightly batch: compare internal ledger against PSP settlement report. Flag any discrepancy for manual review. Financial systems drift — reconciliation catches what the application logic missed.
- Failure mode to name. PSP call times out — did the charge happen? Never assume timeout = failure. Query PSP by idempotency key for status. If PSP confirms: mark COMPLETED. If not found: retry. If ambiguous: hold payment in PENDING and alert ops.
Tradeoffs
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
check if the payment already exists before processing. At 10K TPS with network timeouts, you can't reliably distinguish "never received" from "received and failed" without an idempotency key
client generates UUID before sending any payment request. Server stores the key with status=PENDING before calling the PSP. On retry: find PENDING key → query PSP by the same idempotency key → update to COMPLETED or FAILED
sequence detail: storing the key AFTER the PSP call is a critical mistake. If the process crashes between PSP success and DB write, the retry has no key to find — it creates a new charge. Store the key BEFORE the PSP call unconditionally. On retry: find PENDING key, call PSP with same idempotency key (PSP deduplicates on its end too), update status. The entire chain is idempotent end-to-end. This sequence is the single most important correctness detail in payment system design
after recording the payment in the DB, make an HTTP call to deliver the webhook. If the process crashes between DB commit and HTTP call, the webhook is lost. The merchant never learns about the payment
outbox pattern. In the same ACID transaction that records the payment: INSERT INTO outbox (event_type, payload, status). Either both the payment and the outbox entry commit, or neither does. An async worker polls the outbox for PENDING entries and delivers them with retries and exponential backoff
idempotency for webhooks: the webhook payload must include an idempotency key. The merchant's endpoint may receive the same webhook multiple times (delivery retry after a timeout). If the merchant's system isn't idempotent, a payment can be processed twice on their end. Document this contract explicitly: webhook delivery is at-least-once, merchant endpoints must be idempotent
use a distributed transaction (2PC) across shards to atomically debit the buyer and credit the seller. 2PC is slow, blocks resources during the coordinator phase, and is prone to blocking on coordinator failure
saga pattern. A saga is a sequence of local transactions, each with a compensating transaction on failure. Payment saga: (1) RESERVE buyer funds (local ACID transaction), (2) CREDIT seller, (3) CAPTURE buyer reservation. On failure at step 2: run compensating transaction for step 1 (release reservation)
durability: the saga state machine must be stored durably. If the saga orchestrator crashes mid-saga, it must resume from the last committed step on restart — not restart from the beginning (which would double-charge). Store saga state in PostgreSQL with the current step and status. On restart: read uncommitted sagas, resume from last committed step. Sagas are eventually consistent — the DB may briefly be in an intermediate state, but all failures are handled gracefully by the compensating transactions
Why the deep dives connect to the scaling problem: "High write throughput, strict financial correctness, unreliable external dependencies." Each deep dive addresses one constraint.
Interview script
Whiteboard
+--------------------+
| Merchant Backend |
| uses API keys |
+---------+----------+
|
| HTTPS
v
+-------------+ +-------+--------+
| Customer | | API Gateway |
| Browser | | auth, routing, |
| checkout UI | | rate limiting |
+------+------+ +---+---------+---+
| | |
| card entry via | |
| hosted iframe / SDK | |
v | |
+------+-----------------------+ |
| Secure Payment SDK / iFrame | |
| card data goes to processor | |
+--------------+---------------+ |
| |
| encrypted card data |
| |
v v
+--------+---------+ +--------+---------+
| Transaction | | PaymentIntent |
| Service | | Service |
| creates charge | | create/read |
| records | | payment intent |
+----+--------+----+ +----+--------+----+
| | | |
| +----------------+ |
| read/write |
v v
+---------------------------------------------+
| Operational Database |
| merchants, payment_intents, transactions, |
| attempts, statuses |
+-------------------+-------------------------+
|
| CDC from DB log
v
+--------+---------+
| Kafka / Event |
| Stream |
| immutable events |
+---+----+----+----+
| | |
| | |
| | +------------------+
| | |
| v v
| +---------+ +-------------+
| | Audit | | Webhook |
| | Service | | Service |
| | history | | notify |
| +----+----+ | merchants |
| | +------+------+
| | |
| v | HTTPS POST
| +---------+ v
| | Cold | +-------------+
| | Storage | | Merchant |
| | S3 etc | | Webhook URL |
| +---------+ +-------------+
|
v
+-------+--------+
| Reconciliation |
| Service |
| resolves |
| timeouts |
+-------+--------+
|
| query status / batch files
v
.---------------------------------------.
| External Payment Networks and Banks |
| Visa, Mastercard, issuing banks |
'---------------------------------------'The mental model is simple. PaymentIntent Service manages the customer payment lifecycle, Transaction Service talks to the outside payment world, and the database plus CDC plus Kafka gives you a durable history so you do not lose money movement events.
If you are drawing this in an interview, you can start with just five boxes. Merchant, API Gateway, PaymentIntent Service, Transaction Service, Database. Then add CDC, Kafka, Reconciliation, and Webhooks only if the interviewer pushes on durability or async safety.
batch metrics
decouple ingest
col+time-partitioned
poll 30–60s
retries · resolve
downsample older data
Collect and use system health data at very large scale. Ingest measurements, store as time series, query in dashboards, and trigger alerts.
The hard part: huge write volume, fast queries across long time ranges, reliable alerting, and cardinality explosion.
The hardest part is cardinality explosion. In a metrics system, every unique combination of metric name and labels creates a new time series, so a few extra labels can turn one metric into millions of series very quickly.
That causes three scaling problems. First, ingestion gets expensive because the system is not just appending values. It also has to track metadata and indexes for huge numbers of series. Second, queries get slower because dashboards often need to scan and aggregate across many series over long time ranges. Third, alerts add pressure because they need fresh enough data and reliable evaluation even while the write path is constantly busy.
A good short interview answer is this. Metrics Monitoring is hard to scale because it combines a massive continuous write stream, expensive time range queries, and exploding series count from labels, all while the system itself needs to stay available during incidents.
- Ingest path is write-heavy. Agents batch metrics. Kafka buffers spikes and decouples ingestion from storage.
- TSDB for storage. Metrics are append-only, queried by time range. TSDB is the right default.
- Query path is read-heavy. Rollups + caching make dashboards fast. Rollups: raw 24h → 1min 30d → 1hr forever.
- Alert path must be reliable. Polling every 30–60s is the simple default. Dashboards can be stale. Alerts must not be lost.
- Cardinality explosion. Each unique metric + label combination = a new time series. A label like user_id creates billions of series. Enforce label allowlists and caps.
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
accept all metrics with any label combinations — flexibility is good, users know what they need
cardinality enforcement at ingestion. Each unique combination of metric name + label values creates a new time series. One developer adds user_id to a request_latency metric: 10M users × 1 metric = 10M new series overnight. TSDB memory exhausted, query performance collapses. Hard cap per metric (10K series), alert at 1K series, reject above cap with an actionable error
label allowlist: developers define allowed labels per metric at registration time. user_id is not in the allowlist by default — it requires explicit review. Provide a cardinality dashboard so developers can see the impact of their instrumentation choices before hitting the cap. The allowlist is the proactive control; the hard cap is the safety net
store raw metrics at 1-second resolution forever, query from raw data for all dashboard requests. A 30-day dashboard chart at 1-second resolution = 2.6M data points per metric per host. At 500K hosts, this is 1.3 trillion data points per query — impossible
multi-resolution rollup architecture. Raw (1s): retain 24 hours. 1-minute rollup: retain 30 days. 1-hour rollup: retain forever. Dashboard query routing: time range > 24h → 1-minute rollups; > 30 days → 1-hour rollups. A 30-day chart at 1-min resolution = 1,440 points vs 2.6M raw — 1800× fewer data points, no visible chart difference at typical widths
implementation: TimescaleDB continuous aggregates or InfluxDB tasks compute rollups automatically on insert. The rollup is always up-to-date without a separate batch job. ClickHouse for the rollup serving layer: columnar compression and SIMD-accelerated GROUP BY queries make analytical aggregations over millions of series fast
run alert evaluation as a query against the same TSDB that serves dashboards. During an incident, dashboard traffic spikes 3×. Alert evaluation competes for TSDB resources and gets delayed precisely when alerts are most critical
dedicate separate compute for alert evaluation — separate Kafka consumer group, separate TSDB read replicas, reserved CPU quota. Alert evaluation must never compete with dashboard queries
s: (1) alert evaluation falls behind (Kafka consumer lag) → autoscale alert evaluation workers, alert at >30s lag; (2) TSDB replica goes down → circuit breaker switches to alternate replica; (3) flapping alerts — cooldown periods (alert must be in firing state for N consecutive evaluations before paging) prevent noise during unstable incidents. Monitor the monitoring system: alert evaluation latency is itself a first-class SLA metric
Why the deep dives connect to the scaling problem: "Massive write stream, expensive time-range queries, exploding series count." Each deep dive addresses one dimension.
+----------------------+
| Users / Engineers |
+----------+-----------+
|
v
+----------------------+
| Dashboard / Query UI |
+----------+-----------+
|
v
+----------------------+
| Query Service |
| parse DSL, auth, |
| cache, query split |
+----+------------+----+
| |
cache hit | | query raw or rollups
v v
+---------+ +-------------------+
| Redis | | Time Series DB |
| Cache | | raw + rollups |
+---------+ | sharded + replica |
+---------+---------+
^
|
+---------+---------+
| Storage Consumers |
| batch writes |
+---------+---------+
^
|
+-------------+ +-------------------+ |
| Servers and | ---> | Local Agent / | ---> |
| Services | | Collector | |
| emit metrics| | buffer + batch | |
+-------------+ +---------+---------+ |
| |
v |
+-------------------------+
| Ingestion Service |
| validate, normalize, |
| auth, rate limit |
+-----+-------------+-----+
| |
| v
| +----------------------+
| | Cardinality Guard |
| | policy check |
| | label allowlist |
| +----+------------+----+
| | |
| | v
| | +-----------+
| | | Postgres |
| | | Policies |
| | | Alert cfg |
| | +-----------+
| v
| +-----------+
| | Redis |
| | series set|
| | counters |
| +-----------+
|
v
+----------------------+
| Kafka |
| durable buffer |
| partitioned stream |
+----+-------------+---+
| |
| |
| +----------------------+
| |
v v
+----------------------+ +----------------------+
| Storage Consumers | | Alert Evaluator |
| write to TSDB | | poll rules, query |
+----------------------+ | TSDB every 30 to 60s |
+----------+-----------+
|
v
+----------------------+
| Alert Events |
| firing or resolved |
+----------+-----------+
|
v
+----------------------+
| Notification Service |
| dedupe, grouping, |
| silence, escalation |
+----+----------+------+
| |
| |
v v
+---------+ +----------+
| Slack | | PagerDuty|
+---------+ +----------+
|
v
+------+
|Email |
+------+The main story is ingest, buffer, store, query, then alert. If you are drawing this in an interview, I would keep the first pass even simpler with agents, ingestion, Kafka, time-series DB, query service, alert evaluator, and notification service. Then add cardinality control, cache, and rollups only if the interviewer pushes on scale or latency.
Problem
Collect and use system health data at very large scale. Ingest measurements, store as time series, query in dashboards, and trigger alerts.
The hard part: huge write volume, fast queries across long time ranges, reliable alerting, and cardinality explosion.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
The hardest part is cardinality explosion. In a metrics system, every unique combination of metric name and labels creates a new time series, so a few extra labels can turn one metric into millions of series very quickly.
That causes three scaling problems. First, ingestion gets expensive because the system is not just appending values. It also has to track metadata and indexes for huge numbers of series. Second, queries get slower because dashboards often need to scan and aggregate across many series over long time ranges. Third, alerts add pressure because they need fresh enough data and reliable evaluation even while the write path is constantly busy.
A good short interview answer is this. Metrics Monitoring is hard to scale because it combines a massive continuous write stream, expensive time range queries, and exploding series count from labels, all while the system itself needs to stay available during incidents.
Key points
- Ingest path is write-heavy. Agents batch metrics. Kafka buffers spikes and decouples ingestion from storage.
- TSDB for storage. Metrics are append-only, queried by time range. TSDB is the right default.
- Query path is read-heavy. Rollups + caching make dashboards fast. Rollups: raw 24h → 1min 30d → 1hr forever.
- Alert path must be reliable. Polling every 30–60s is the simple default. Dashboards can be stale. Alerts must not be lost.
- Cardinality explosion. Each unique metric + label combination = a new time series. A label like user_id creates billions of series. Enforce label allowlists and caps.
Tradeoffs
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
accept all metrics with any label combinations — flexibility is good, users know what they need
cardinality enforcement at ingestion. Each unique combination of metric name + label values creates a new time series. One developer adds user_id to a request_latency metric: 10M users × 1 metric = 10M new series overnight. TSDB memory exhausted, query performance collapses. Hard cap per metric (10K series), alert at 1K series, reject above cap with an actionable error
label allowlist: developers define allowed labels per metric at registration time. user_id is not in the allowlist by default — it requires explicit review. Provide a cardinality dashboard so developers can see the impact of their instrumentation choices before hitting the cap. The allowlist is the proactive control; the hard cap is the safety net
store raw metrics at 1-second resolution forever, query from raw data for all dashboard requests. A 30-day dashboard chart at 1-second resolution = 2.6M data points per metric per host. At 500K hosts, this is 1.3 trillion data points per query — impossible
multi-resolution rollup architecture. Raw (1s): retain 24 hours. 1-minute rollup: retain 30 days. 1-hour rollup: retain forever. Dashboard query routing: time range > 24h → 1-minute rollups; > 30 days → 1-hour rollups. A 30-day chart at 1-min resolution = 1,440 points vs 2.6M raw — 1800× fewer data points, no visible chart difference at typical widths
implementation: TimescaleDB continuous aggregates or InfluxDB tasks compute rollups automatically on insert. The rollup is always up-to-date without a separate batch job. ClickHouse for the rollup serving layer: columnar compression and SIMD-accelerated GROUP BY queries make analytical aggregations over millions of series fast
run alert evaluation as a query against the same TSDB that serves dashboards. During an incident, dashboard traffic spikes 3×. Alert evaluation competes for TSDB resources and gets delayed precisely when alerts are most critical
dedicate separate compute for alert evaluation — separate Kafka consumer group, separate TSDB read replicas, reserved CPU quota. Alert evaluation must never compete with dashboard queries
s: (1) alert evaluation falls behind (Kafka consumer lag) → autoscale alert evaluation workers, alert at >30s lag; (2) TSDB replica goes down → circuit breaker switches to alternate replica; (3) flapping alerts — cooldown periods (alert must be in firing state for N consecutive evaluations before paging) prevent noise during unstable incidents. Monitor the monitoring system: alert evaluation latency is itself a first-class SLA metric
Why the deep dives connect to the scaling problem: "Massive write stream, expensive time-range queries, exploding series count." Each deep dive addresses one dimension.
Interview script
Whiteboard
+----------------------+
| Users / Engineers |
+----------+-----------+
|
v
+----------------------+
| Dashboard / Query UI |
+----------+-----------+
|
v
+----------------------+
| Query Service |
| parse DSL, auth, |
| cache, query split |
+----+------------+----+
| |
cache hit | | query raw or rollups
v v
+---------+ +-------------------+
| Redis | | Time Series DB |
| Cache | | raw + rollups |
+---------+ | sharded + replica |
+---------+---------+
^
|
+---------+---------+
| Storage Consumers |
| batch writes |
+---------+---------+
^
|
+-------------+ +-------------------+ |
| Servers and | ---> | Local Agent / | ---> |
| Services | | Collector | |
| emit metrics| | buffer + batch | |
+-------------+ +---------+---------+ |
| |
v |
+-------------------------+
| Ingestion Service |
| validate, normalize, |
| auth, rate limit |
+-----+-------------+-----+
| |
| v
| +----------------------+
| | Cardinality Guard |
| | policy check |
| | label allowlist |
| +----+------------+----+
| | |
| | v
| | +-----------+
| | | Postgres |
| | | Policies |
| | | Alert cfg |
| | +-----------+
| v
| +-----------+
| | Redis |
| | series set|
| | counters |
| +-----------+
|
v
+----------------------+
| Kafka |
| durable buffer |
| partitioned stream |
+----+-------------+---+
| |
| |
| +----------------------+
| |
v v
+----------------------+ +----------------------+
| Storage Consumers | | Alert Evaluator |
| write to TSDB | | poll rules, query |
+----------------------+ | TSDB every 30 to 60s |
+----------+-----------+
|
v
+----------------------+
| Alert Events |
| firing or resolved |
+----------+-----------+
|
v
+----------------------+
| Notification Service |
| dedupe, grouping, |
| silence, escalation |
+----+----------+------+
| |
| |
v v
+---------+ +----------+
| Slack | | PagerDuty|
+---------+ +----------+
|
v
+------+
|Email |
+------+The main story is ingest, buffer, store, query, then alert. If you are drawing this in an interview, I would keep the first pass even simpler with agents, ingestion, Kafka, time-series DB, query service, alert evaluator, and notification service. Then add cardinality control, cache, and rollups only if the interviewer pushes on scale or latency.
replicated
partition
offset
Durable, high-throughput pub/sub log that decouples producers and consumers, scales horizontally, and replays history.
Hard parts: partition key design, consumer lag, and exactly-once vs at-least-once semantics.
Kafka scales by adding partitions and brokers. Limits: partition count planning, consumer lag under slow workers, hot keys.
- Partition by key. Same key → same partition preserves ordering per entity.
- Consumer group scaling. Max parallelism = partition count. More consumers than partitions sit idle.
- Replication factor 3. Leader + followers. ISR replicas must ack before commit (configurable).
- Retention. Log retained days/weeks — consumers can replay or catch up.
- At-least-once default. Commit offset after process. Idempotent consumers handle duplicates.
- Dead letter topic. Poison messages after N failures — do not block partition.
- Monitor consumer lag. Lag = high-priority alert. Autoscale consumers on lag.
Wrong key (constant) → one hot partition. Right key (user_id, order_id) spreads load and preserves per-entity order
Oversimplify partition key design — name one component, skip failure modes and metrics.
Wrong key (constant) → one hot partition. Right key (user_id, order_id) spreads load and preserves per-entity order
Name metric + revisit trigger when they push depth.
Adding consumer triggers rebalance — brief pause. Use cooperative sticky assignor to minimize disruption. Max consumers = partitions
Oversimplify consumer groups and rebalancing — name one component, skip failure modes and metrics.
Adding consumer triggers rebalance — brief pause. Use cooperative sticky assignor to minimize disruption. Max consumers = partitions
Name metric + revisit trigger when they push depth.
min.insync.replicas=2 with acks=all prevents data loss on broker failure. Unclean leader election trades availability for loss risk — avoid for financial topics
UUID v4 everywhere — collisions are negligible.
min.insync.replicas=2 with acks=all prevents data loss on broker failure. Unclean leader election trades availability for loss risk — avoid for financial topics
Name metric + revisit trigger when they push depth.
After 3 failures route to DLQ. Skip bad message so partition progresses. Alert on DLQ depth
Oversimplify handling poison pills — name one component, skip failure modes and metrics.
After 3 failures route to DLQ. Skip bad message so partition progresses. Alert on DLQ depth
Name metric + revisit trigger when they push depth.
Producers -> Kafka Topic (P0..Pn) -> Consumer Group -> Workers
\-> replicas in ISRDraw partitions and consumer group. Mention ISR for durability.
Problem
Durable, high-throughput pub/sub log that decouples producers and consumers, scales horizontally, and replays history.
Hard parts: partition key design, consumer lag, and exactly-once vs at-least-once semantics.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
Kafka scales by adding partitions and brokers. Limits: partition count planning, consumer lag under slow workers, hot keys.
Key points
- Partition by key. Same key → same partition preserves ordering per entity.
- Consumer group scaling. Max parallelism = partition count. More consumers than partitions sit idle.
- Replication factor 3. Leader + followers. ISR replicas must ack before commit (configurable).
- Retention. Log retained days/weeks — consumers can replay or catch up.
- At-least-once default. Commit offset after process. Idempotent consumers handle duplicates.
- Dead letter topic. Poison messages after N failures — do not block partition.
- Monitor consumer lag. Lag = high-priority alert. Autoscale consumers on lag.
Tradeoffs
Deep dives
Wrong key (constant) → one hot partition. Right key (user_id, order_id) spreads load and preserves per-entity order
Oversimplify partition key design — name one component, skip failure modes and metrics.
Wrong key (constant) → one hot partition. Right key (user_id, order_id) spreads load and preserves per-entity order
Name metric + revisit trigger when they push depth.
Adding consumer triggers rebalance — brief pause. Use cooperative sticky assignor to minimize disruption. Max consumers = partitions
Oversimplify consumer groups and rebalancing — name one component, skip failure modes and metrics.
Adding consumer triggers rebalance — brief pause. Use cooperative sticky assignor to minimize disruption. Max consumers = partitions
Name metric + revisit trigger when they push depth.
min.insync.replicas=2 with acks=all prevents data loss on broker failure. Unclean leader election trades availability for loss risk — avoid for financial topics
UUID v4 everywhere — collisions are negligible.
min.insync.replicas=2 with acks=all prevents data loss on broker failure. Unclean leader election trades availability for loss risk — avoid for financial topics
Name metric + revisit trigger when they push depth.
After 3 failures route to DLQ. Skip bad message so partition progresses. Alert on DLQ depth
Oversimplify handling poison pills — name one component, skip failure modes and metrics.
After 3 failures route to DLQ. Skip bad message so partition progresses. Alert on DLQ depth
Name metric + revisit trigger when they push depth.
Interview script
Whiteboard
Producers -> Kafka Topic (P0..Pn) -> Consumer Group -> Workers
\-> replicas in ISRDraw partitions and consumer group. Mention ISR for durability.
Build Dynamo-style highly available KV store: partition data across nodes, replicate for durability, remain available during partitions.
Tradeoff: availability + partition tolerance vs strong consistency.
KV store scales by adding nodes to the ring. Pain points: hot keys, quorum latency, conflict resolution complexity.
- Consistent hashing + vnodes. Even load distribution; minimal remapping on node add/remove.
- Replication factor N=3. Write to N nodes. Tolerate N-1 failures with proper quorum.
- Quorum reads/writes. W writes, R reads, R+W>N gives consistency guarantee.
- Vector clocks. Detect concurrent updates; client or system resolves siblings.
- Sloppy quorum + hinted handoff. Write to W healthy nodes even if primary down; hand off when node returns.
- Anti-entropy. Merkle tree sync repairs divergence between replicas.
- CAP choice. AP system — available during partition, eventual consistency.
Without vnodes, ring imbalance 3×. 150 vnodes per physical node smooths distribution. Node add/remove remaps only adjacent key ranges
Oversimplify consistent hashing and virtual nodes — name one component, skip failure modes and metrics.
Without vnodes, ring imbalance 3×. 150 vnodes per physical node smooths distribution. Node add/remove remaps only adjacent key ranges
Name metric + revisit trigger when they push depth.
N=3, W=2, R=2 → R+W>N guarantees read sees latest write. W=1, R=1 fastest but stale reads possible. Tune per use case
Oversimplify quorum math — name one component, skip failure modes and metrics.
N=3, W=2, R=2 → R+W>N guarantees read sees latest write. W=1, R=1 fastest but stale reads possible. Tune per use case
Name metric + revisit trigger when they push depth.
Node down: write to alternative node with hint. On recovery, hand off data. Background Merkle tree comparison finds drift
Oversimplify failure handling — name one component, skip failure modes and metrics.
Node down: write to alternative node with hint. On recovery, hand off data. Background Merkle tree comparison finds drift
Name metric + revisit trigger when they push depth.
Concurrent puts create sibling versions. Client reads all siblings, merges (e.g., cart union), writes resolved version
Oversimplify conflict resolution — name one component, skip failure modes and metrics.
Concurrent puts create sibling versions. Client reads all siblings, merges (e.g., cart union), writes resolved version
Name metric + revisit trigger when they push depth.
Client -> Coordinator -> hash ring -> Replica nodes (N=3)
W writes, R reads, gossip membershipRing diagram + quorum numbers on whiteboard.
Problem
Build Dynamo-style highly available KV store: partition data across nodes, replicate for durability, remain available during partitions.
Tradeoff: availability + partition tolerance vs strong consistency.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
KV store scales by adding nodes to the ring. Pain points: hot keys, quorum latency, conflict resolution complexity.
Key points
- Consistent hashing + vnodes. Even load distribution; minimal remapping on node add/remove.
- Replication factor N=3. Write to N nodes. Tolerate N-1 failures with proper quorum.
- Quorum reads/writes. W writes, R reads, R+W>N gives consistency guarantee.
- Vector clocks. Detect concurrent updates; client or system resolves siblings.
- Sloppy quorum + hinted handoff. Write to W healthy nodes even if primary down; hand off when node returns.
- Anti-entropy. Merkle tree sync repairs divergence between replicas.
- CAP choice. AP system — available during partition, eventual consistency.
Tradeoffs
Deep dives
Without vnodes, ring imbalance 3×. 150 vnodes per physical node smooths distribution. Node add/remove remaps only adjacent key ranges
Oversimplify consistent hashing and virtual nodes — name one component, skip failure modes and metrics.
Without vnodes, ring imbalance 3×. 150 vnodes per physical node smooths distribution. Node add/remove remaps only adjacent key ranges
Name metric + revisit trigger when they push depth.
N=3, W=2, R=2 → R+W>N guarantees read sees latest write. W=1, R=1 fastest but stale reads possible. Tune per use case
Oversimplify quorum math — name one component, skip failure modes and metrics.
N=3, W=2, R=2 → R+W>N guarantees read sees latest write. W=1, R=1 fastest but stale reads possible. Tune per use case
Name metric + revisit trigger when they push depth.
Node down: write to alternative node with hint. On recovery, hand off data. Background Merkle tree comparison finds drift
Oversimplify failure handling — name one component, skip failure modes and metrics.
Node down: write to alternative node with hint. On recovery, hand off data. Background Merkle tree comparison finds drift
Name metric + revisit trigger when they push depth.
Concurrent puts create sibling versions. Client reads all siblings, merges (e.g., cart union), writes resolved version
Oversimplify conflict resolution — name one component, skip failure modes and metrics.
Concurrent puts create sibling versions. Client reads all siblings, merges (e.g., cart union), writes resolved version
Name metric + revisit trigger when they push depth.
Interview script
Whiteboard
Client -> Coordinator -> hash ring -> Replica nodes (N=3)
W writes, R reads, gossip membershipRing diagram + quorum numbers on whiteboard.
Show users which friends are physically nearby in real time.
Hard parts: high-frequency location updates, efficient geo queries, and privacy/battery constraints.
High-frequency ephemeral writes and geo indexing are the pain points — not friend graph size.
- Ephemeral location in Redis. 2M users × update/30s = 67K writes/s — Redis yes, PostgreSQL no.
- Geohash cells. Map lat/long to cell. Query cell + 8 neighbors covers edge cases.
- Friend graph in PG. Friendships durable in PostgreSQL. Intersect geo results with friend list.
- Push on proximity. WebSocket notify when friend enters radius — not poll.
- Privacy controls. Ghost mode, precision reduction, sharing window.
- Battery. Adaptive update interval: stationary → 5 min, moving → 30s.
- Stale location TTL. EXPIRE location keys 5 min — no ghost users on map.
67K GEOADD/s is Redis-comfortable. Never persist every fix to PG. TTL keys expire stale users
Oversimplify write path at scale — name one component, skip failure modes and metrics.
67K GEOADD/s is Redis-comfortable. Never persist every fix to PG. TTL keys expire stale users
Name metric + revisit trigger when they push depth.
Compute user cell. Fetch users in cell + 8 neighbors. Filter by haversine distance < R. Intersect with friend IDs from PG/cache
Oversimplify query algorithm — name one component, skip failure modes and metrics.
Compute user cell. Fetch users in cell + 8 neighbors. Filter by haversine distance < R. Intersect with friend IDs from PG/cache
Name metric + revisit trigger when they push depth.
Reduce precision to 5-char geohash (~5km) by default. Ghost mode deletes Redis entry immediately
Oversimplify privacy and precision — name one component, skip failure modes and metrics.
Reduce precision to 5-char geohash (~5km) by default. Ghost mode deletes Redis entry immediately
Name metric + revisit trigger when they push depth.
Do not push every 30s if friend still nearby. State machine: ENTERED_NEARBY → INSIDE → EXITED. Push only on transitions
Retry until delivery succeeds — duplicates are rare.
Do not push every 30s if friend still nearby. State machine: ENTERED_NEARBY → INSIDE → EXITED. Push only on transitions
Name metric + revisit trigger when they push depth.
GPS -> Location Svc -> Redis GEO (per geohash cell) Friend list <- PostgreSQL Matcher -> intersect -> WebSocket push
Two stores: ephemeral geo in Redis, social graph in PG.
Problem
Show users which friends are physically nearby in real time.
Hard parts: high-frequency location updates, efficient geo queries, and privacy/battery constraints.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
High-frequency ephemeral writes and geo indexing are the pain points — not friend graph size.
Key points
- Ephemeral location in Redis. 2M users × update/30s = 67K writes/s — Redis yes, PostgreSQL no.
- Geohash cells. Map lat/long to cell. Query cell + 8 neighbors covers edge cases.
- Friend graph in PG. Friendships durable in PostgreSQL. Intersect geo results with friend list.
- Push on proximity. WebSocket notify when friend enters radius — not poll.
- Privacy controls. Ghost mode, precision reduction, sharing window.
- Battery. Adaptive update interval: stationary → 5 min, moving → 30s.
- Stale location TTL. EXPIRE location keys 5 min — no ghost users on map.
Tradeoffs
Deep dives
67K GEOADD/s is Redis-comfortable. Never persist every fix to PG. TTL keys expire stale users
Oversimplify write path at scale — name one component, skip failure modes and metrics.
67K GEOADD/s is Redis-comfortable. Never persist every fix to PG. TTL keys expire stale users
Name metric + revisit trigger when they push depth.
Compute user cell. Fetch users in cell + 8 neighbors. Filter by haversine distance < R. Intersect with friend IDs from PG/cache
Oversimplify query algorithm — name one component, skip failure modes and metrics.
Compute user cell. Fetch users in cell + 8 neighbors. Filter by haversine distance < R. Intersect with friend IDs from PG/cache
Name metric + revisit trigger when they push depth.
Reduce precision to 5-char geohash (~5km) by default. Ghost mode deletes Redis entry immediately
Oversimplify privacy and precision — name one component, skip failure modes and metrics.
Reduce precision to 5-char geohash (~5km) by default. Ghost mode deletes Redis entry immediately
Name metric + revisit trigger when they push depth.
Do not push every 30s if friend still nearby. State machine: ENTERED_NEARBY → INSIDE → EXITED. Push only on transitions
Retry until delivery succeeds — duplicates are rare.
Do not push every 30s if friend still nearby. State machine: ENTERED_NEARBY → INSIDE → EXITED. Push only on transitions
Name metric + revisit trigger when they push depth.
Interview script
Whiteboard
GPS -> Location Svc -> Redis GEO (per geohash cell) Friend list <- PostgreSQL Matcher -> intersect -> WebSocket push
Two stores: ephemeral geo in Redis, social graph in PG.
Build a global maps platform: render maps fast worldwide, search places, and compute driving directions at scale.
Hard parts: petabytes of tiles, sub-100ms pan/zoom, and routing on a graph with hundreds of millions of edges.
Petabyte-scale immutable tiles and global CDN hit ratio dominate — routing is compute-heavy but smaller QPS.
- Tile pyramid. Zoom level z has 4^z tiles globally. Pre-generate popular regions; on-demand for long tail.
- CDN-first. Tiles are immutable — Cache-Control: max-age=31536000. CDN handles 99% of map traffic.
- Geospatial search. POIs indexed in ES with geo_point. Query: text match + geo_distance filter + popularity boost.
- Routing graph. Preprocessed road network offline. Online: bidirectional Dijkstra or contraction hierarchies.
- Live traffic. Traffic overlay is dynamic — separate tile layer or client-side vector update, not baked into base tiles.
- Personalization out of scope. Unless asked: saved places, ads, Street View capture pipeline.
generate tiles per request
pre-render pyramid, store in S3, serve via CDN. Staff+: invalidation only for traffic/incident overlays
Name the metric you'd alert on and when you'd revisit this design.
Text relevance × distance decay × popularity. Geo filter first to shrink candidate set
SELECT * WHERE column LIKE '%query%'.
Text relevance × distance decay × popularity. Geo filter first to shrink candidate set
Name metric + revisit trigger when they push depth.
Graph partitioned by region. Highway hierarchy: coarse graph for long distances, refine locally
Oversimplify routing at scale — name one component, skip failure modes and metrics.
Graph partitioned by region. Highway hierarchy: coarse graph for long distances, refine locally
Name metric + revisit trigger when they push depth.
Probe GPS stream → aggregate speeds per road segment → publish traffic layer every 2–5 min
Oversimplify fresh traffic data — name one component, skip failure modes and metrics.
Probe GPS stream → aggregate speeds per road segment → publish traffic layer every 2–5 min
Name metric + revisit trigger when they push depth.
Client -> CDN -> S3 tiles (base map) Client -> API -> ES (POI search) Client -> Routing svc -> Graph shards (CH / hub labels)
Separate read paths for tiles, search, and routing.
Problem
Build a global maps platform: render maps fast worldwide, search places, and compute driving directions at scale.
Hard parts: petabytes of tiles, sub-100ms pan/zoom, and routing on a graph with hundreds of millions of edges.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
Petabyte-scale immutable tiles and global CDN hit ratio dominate — routing is compute-heavy but smaller QPS.
Key points
- Tile pyramid. Zoom level z has 4^z tiles globally. Pre-generate popular regions; on-demand for long tail.
- CDN-first. Tiles are immutable — Cache-Control: max-age=31536000. CDN handles 99% of map traffic.
- Geospatial search. POIs indexed in ES with geo_point. Query: text match + geo_distance filter + popularity boost.
- Routing graph. Preprocessed road network offline. Online: bidirectional Dijkstra or contraction hierarchies.
- Live traffic. Traffic overlay is dynamic — separate tile layer or client-side vector update, not baked into base tiles.
- Personalization out of scope. Unless asked: saved places, ads, Street View capture pipeline.
Tradeoffs
Deep dives
generate tiles per request
pre-render pyramid, store in S3, serve via CDN. Staff+: invalidation only for traffic/incident overlays
Name the metric you'd alert on and when you'd revisit this design.
Text relevance × distance decay × popularity. Geo filter first to shrink candidate set
SELECT * WHERE column LIKE '%query%'.
Text relevance × distance decay × popularity. Geo filter first to shrink candidate set
Name metric + revisit trigger when they push depth.
Graph partitioned by region. Highway hierarchy: coarse graph for long distances, refine locally
Oversimplify routing at scale — name one component, skip failure modes and metrics.
Graph partitioned by region. Highway hierarchy: coarse graph for long distances, refine locally
Name metric + revisit trigger when they push depth.
Probe GPS stream → aggregate speeds per road segment → publish traffic layer every 2–5 min
Oversimplify fresh traffic data — name one component, skip failure modes and metrics.
Probe GPS stream → aggregate speeds per road segment → publish traffic layer every 2–5 min
Name metric + revisit trigger when they push depth.
Interview script
Whiteboard
Client -> CDN -> S3 tiles (base map) Client -> API -> ES (POI search) Client -> Routing svc -> Graph shards (CH / hub labels)
Separate read paths for tiles, search, and routing.
Design webmail at Gmail scale: receive, store, search, and send billions of emails with strong per-user consistency.
Hard parts: storage per user, full-text search, and reliable SMTP delivery.
Billions of messages/day with large attachments — sharding and blob offload are mandatory.
- Shard by user_id. All of a user's mail on one shard — simplifies inbox listing and ACID per mailbox.
- Blob for attachments. Message metadata in Cassandra; attachment bytes in S3/HDFS.
- Async search index. Kafka mail event → ES indexer. Search slightly behind inbox — acceptable.
- Spam at ingress. Score before storage. Quarantine bucket for suspicious mail.
- SMTP outbound queue. Retry with exponential backoff. DKIM/SPF signing per domain.
- Deletion/tombstones. Soft delete + async purge from index and blob store.
one row per email in SQL
Cassandra partition key = user_id, cluster key = timestamp+id. Staff+: separate hot (inbox) and cold (archive) tiers
Name the metric you'd alert on and when you'd revisit this design.
Inverted index per user or global with user_id filter. Reindex pipeline from mail log for recovery
Rebuild the full index nightly — no incremental updates.
Inverted index per user or global with user_id filter. Reindex pipeline from mail log for recovery
Name metric + revisit trigger when they push depth.
Outbound queue in Kafka. Multiple MX retries. Bounce handling updates recipient reputation
Oversimplify smtp reliability — name one component, skip failure modes and metrics.
Outbound queue in Kafka. Multiple MX retries. Bounce handling updates recipient reputation
Name metric + revisit trigger when they push depth.
Feature extraction at edge. ML model ensemble. User feedback loop for false positives
Query the database on every feed request.
Feature extraction at edge. ML model ensemble. User feedback loop for false positives
Name metric + revisit trigger when they push depth.
SMTP -> Ingest -> Spam -> Cassandra (user shard) -> API -> Client
-> Blob (attachments)
-> Kafka -> ES indexerUser shard is source of truth; search is derived.
Problem
Design webmail at Gmail scale: receive, store, search, and send billions of emails with strong per-user consistency.
Hard parts: storage per user, full-text search, and reliable SMTP delivery.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
Billions of messages/day with large attachments — sharding and blob offload are mandatory.
Key points
- Shard by user_id. All of a user's mail on one shard — simplifies inbox listing and ACID per mailbox.
- Blob for attachments. Message metadata in Cassandra; attachment bytes in S3/HDFS.
- Async search index. Kafka mail event → ES indexer. Search slightly behind inbox — acceptable.
- Spam at ingress. Score before storage. Quarantine bucket for suspicious mail.
- SMTP outbound queue. Retry with exponential backoff. DKIM/SPF signing per domain.
- Deletion/tombstones. Soft delete + async purge from index and blob store.
Tradeoffs
Deep dives
one row per email in SQL
Cassandra partition key = user_id, cluster key = timestamp+id. Staff+: separate hot (inbox) and cold (archive) tiers
Name the metric you'd alert on and when you'd revisit this design.
Inverted index per user or global with user_id filter. Reindex pipeline from mail log for recovery
Rebuild the full index nightly — no incremental updates.
Inverted index per user or global with user_id filter. Reindex pipeline from mail log for recovery
Name metric + revisit trigger when they push depth.
Outbound queue in Kafka. Multiple MX retries. Bounce handling updates recipient reputation
Oversimplify smtp reliability — name one component, skip failure modes and metrics.
Outbound queue in Kafka. Multiple MX retries. Bounce handling updates recipient reputation
Name metric + revisit trigger when they push depth.
Feature extraction at edge. ML model ensemble. User feedback loop for false positives
Query the database on every feed request.
Feature extraction at edge. ML model ensemble. User feedback loop for false positives
Name metric + revisit trigger when they push depth.
Interview script
Whiteboard
SMTP -> Ingest -> Spam -> Cassandra (user shard) -> API -> Client
-> Blob (attachments)
-> Kafka -> ES indexerUser shard is source of truth; search is derived.
Design a digital wallet: add cards, pay in stores/apps, P2P transfers, with PCI compliance and financial correctness.
Hard parts: tokenization, fraud, and ledger integrity.
Financial correctness and PCI boundaries matter more than raw QPS.
- Tokenization. Replace PAN with device-specific token. HSM generates and stores mapping.
- PCI scope reduction. Merchant handles tokens only. Vault is isolated PCI zone.
- Double-entry ledger. Every transfer = balanced debit/credit events. Immutable log.
- Idempotent payments. client_request_id dedup prevents double tap charges.
- 3DS / biometrics. Step-up auth for high-risk transactions.
- P2P transfers. Internal ledger move before external ACH settlement.
encrypt PAN in DB
HSM generates tokens; PAN never leaves secure enclave
Name the metric you'd alert on and when you'd revisit this design.
Append-only events. Daily balance invariant check. No in-place balance updates
Oversimplify ledger correctness — name one component, skip failure modes and metrics.
Append-only events. Daily balance invariant check. No in-place balance updates
Name metric + revisit trigger when they push depth.
Velocity limits, device fingerprint, ML risk score. Step-up 3DS above threshold
Oversimplify fraud — name one component, skip failure modes and metrics.
Velocity limits, device fingerprint, ML risk score. Step-up 3DS above threshold
Name metric + revisit trigger when they push depth.
Stored cryptogram on secure element. Limited offline spend counter
One global INCR key for all traffic.
Stored cryptogram on secure element. Limited offline spend counter
Name metric + revisit trigger when they push depth.
App -> Wallet API -> Token Vault (HSM)
-> Auth svc -> Payment network
-> Ledger (event log)Vault and ledger are isolated trust boundaries.
Problem
Design a digital wallet: add cards, pay in stores/apps, P2P transfers, with PCI compliance and financial correctness.
Hard parts: tokenization, fraud, and ledger integrity.
Failures
Estimation
Design decisions
Follow-up Q&A
Evolution
Why it's hard to scale
Financial correctness and PCI boundaries matter more than raw QPS.
Key points
- Tokenization. Replace PAN with device-specific token. HSM generates and stores mapping.
- PCI scope reduction. Merchant handles tokens only. Vault is isolated PCI zone.
- Double-entry ledger. Every transfer = balanced debit/credit events. Immutable log.
- Idempotent payments. client_request_id dedup prevents double tap charges.
- 3DS / biometrics. Step-up auth for high-risk transactions.
- P2P transfers. Internal ledger move before external ACH settlement.
Tradeoffs
Deep dives
encrypt PAN in DB
HSM generates tokens; PAN never leaves secure enclave
Name the metric you'd alert on and when you'd revisit this design.
Append-only events. Daily balance invariant check. No in-place balance updates
Oversimplify ledger correctness — name one component, skip failure modes and metrics.
Append-only events. Daily balance invariant check. No in-place balance updates
Name metric + revisit trigger when they push depth.
Velocity limits, device fingerprint, ML risk score. Step-up 3DS above threshold
Oversimplify fraud — name one component, skip failure modes and metrics.
Velocity limits, device fingerprint, ML risk score. Step-up 3DS above threshold
Name metric + revisit trigger when they push depth.
Stored cryptogram on secure element. Limited offline spend counter
One global INCR key for all traffic.
Stored cryptogram on secure element. Limited offline spend counter
Name metric + revisit trigger when they push depth.
Interview script
Whiteboard
App -> Wallet API -> Token Vault (HSM)
-> Auth svc -> Payment network
-> Ledger (event log)Vault and ledger are isolated trust boundaries.