A user submits a long URL. The API server hashes it, encodes seven characters in Base62, and writes the mapping to PostgreSQL and Redis. On redirect the service checks Redis first — reads outnumber writes ~100:1 so the cache absorbs almost all traffic. A cache miss falls back to PG, repopulates Redis, and returns a 302 redirect (not 301) so every click hits the server for analytics.
Problem
Bitly solves short link creation and fast redirection at scale. You take a long URL, store a mapping to a shorter code, and when someone visits the short link the system looks up the original and redirects them.
The real challenges: making short codes unique, keeping redirects fast, and handling a read-heavy workload — links are clicked far more than they are created.
Failures
**Redis goes down**
All redirects fall through to PostgreSQL. At peak traffic this overwhelms PG. Latency spikes from sub-ms to tens of ms.
_Fix:_ Fail open to DB reads. Set PG connection pool high. Add a second Redis replica. Circuit-break if PG > 80% capacity.
**Counter service fails (code generation)**
No new short URLs can be created. Existing redirects still work.
_Fix:_ Counter is a single point of failure. Mitigate with pre-allocated ID ranges per app server (each takes a batch of 1000 IDs). Graceful degradation: queue creation requests.
**Hot link (viral URL)**
Single short code hammers Redis then PG if evicted. One key exhausts connection pool.
_Fix:_ Local in-process cache on each app server for top-N keys. CDN caching for redirect responses (302 with short Cache-Control).
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 500M total URLs, 100M DAU, average user clicks 10 short links/day |
| Read QPS | 100M × 10 / 86400 ≈ 11,600 read QPS |
| Write QPS | 10M new URLs/day / 86400 ≈ 116 write QPS — read:write ratio ~100:1 |
| Storage | 500M rows × 500 bytes (url + metadata) ≈ 250 GB — comfortably single PG node |
| Cache math | Top 20% of URLs get 80% of traffic → cache 100M URLs × 100 bytes ≈ 10 GB Redis — fits easily |
| Verdict | Single PG node fine for writes. Redis absorbs 99%+ of reads. Scale app servers horizontally for redirect serving. |
Design decisions
**Base62 counter over random hash**
→ Global counter + Base62
Guaranteed uniqueness, no collision handling, predictable length (7 chars at 62^7 = 3.5T URLs). Tradeoff: codes are guessable — acceptable for most use cases.
_Revisit when:_ Switch to random hash + collision retry if enumeration is a security concern.
**302 over 301 redirect**
→ 302 Temporary
301 caches permanently in browser — lose analytics, can't expire, can't update destinations. 302 hits server every time giving full control.
_Revisit when:_ Could offer 301 as an opt-in for customers who want to reduce load.
**Redis as cache, PG as source of truth**
→ Cache-aside pattern
Write-through would add latency on every creation. Cache-aside is simpler: write to PG, lazy-populate Redis on first read.
_Revisit when:_ Write-through if cache miss rate ever exceeds 5%.
Follow-up Q&A
**What happens if Redis is completely unavailable?**
Fail open to PostgreSQL. Short-term pain (higher latency) beats failing closed (all redirects return 5xx). Add a circuit breaker so if PG latency exceeds 500ms we start returning 503 with Retry-After.
**How do you handle a single viral URL causing a hot key?**
Two layers: local in-process LRU cache on each app server (top 1000 keys, zero network hops), and optionally a CDN caching 302 responses for a short TTL. The hot key never reaches Redis or PG.
**How would you handle 10× traffic suddenly?**
App servers are stateless — auto-scale horizontally immediately. Redis handles 10× without changes (it's in-memory). PG might need a read replica. The counter service is the only coordination point — pre-allocate large ID batches to reduce contention.
**How do you expire links?**
Store expiry timestamp in PG. On redirect, check expiry — if expired, return 410 Gone and delete from Redis. TTL the Redis key to match expiry time so it auto-evicts.
**How would you support custom aliases?**
Store alias in same URL table with a unique constraint. On creation: check alias not taken (SELECT FOR UPDATE or optimistic retry), store it. Downside: can't guarantee 7-char length for aliases.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: success rate, latency, active users. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: minutes of read unavailability acceptable; rebuild cache from DB. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Single PostgreSQL + single app server. No cache. Counter in DB (SELECT MAX + 1). Handles up to ~1K QPS. Fine for early product.
**v2 — Scale reads** — Add Redis cache. Separate read and write app services. Counter service with pre-allocated ID batches. Handles 50K QPS reads. Deploy globally with regional Redis replicas.
**v3 — Optimize** — CDN for popular redirects. Local in-process cache for viral links. Async analytics pipeline (Kafka → ClickHouse). Custom domain support. Link expiry cleanup job. Handles 500K+ QPS.
Why it's hard to scale
The hard part is that reads explode while writes stay small. A Bitly style system might create URLs slowly, but a single popular link can suddenly cause huge redirect traffic, so you need very fast lookups and a way to survive spikes.
The other tricky part is global uniqueness for short codes. If you scale the write path across many servers, they all need to agree on which code comes next or you risk collisions. So the two main scaling pain points are read traffic on redirects and coordination for code generation.
Key points
- **Scope** — URL in, short URL out. Short URL in, redirect out. Keep analytics and auth out of scope unless asked.
- **Key requirement** — Read-heavy system — optimize redirects first.
- **API** — POST /urls creates a short link. GET /{shortCode} redirects.
- **Data model** — short_code → long_url + created time + optional expiration. short_code is the primary key.
- **Code generation** — Global counter + base62 is the best default. Easy to explain, guarantees uniqueness.
- **Fast reads** — Redis cache in front of DB. Cache first, DB on miss.
- **Redirect** — Return 302 not 301 — keeps control and avoids permanent browser caching.
- **Expiration** — Check expiry on read. Return 410 Gone if expired. Match cache TTL to expiration.
- **Scale** — Stateless app servers behind a load balancer. Split read and write services if needed.
> Memory hook: counter, cache, redirect. Say those three and you have covered the heart of the design.
Tradeoffs
**First** — Counter vs hash. Counter + base62 guarantees uniqueness and is easy to explain. Tradeoff: needs coordination, codes are predictable. Hash is more distributed but requires collision handling.
**Second** — DB only vs cache + DB. Cache + DB is right for a read-heavy system. Tradeoff: extra complexity around misses, eviction, and expired links.
**Third** — 302 vs 301. 302 keeps control server-side. Tradeoff: 301 reduces repeat load but makes expiration harder.
> "I picked the simplest design that meets the requirements. Main tradeoffs: uniqueness vs coordination, speed vs complexity, control vs caching behavior."
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
#### Deep dive 1: Unique short code generation (the core hard problem)
_The scaling pain here is coordination — every server needs a unique code without collision_
> [!CAUTION]
> **🔴 Weak** — Describe hashing and move on
>
> [!WARNING]
> **🟡 Strong** — articulate a progression of approaches and their tradeoffs. Start with MD5/SHA-256 + Base62 truncation: simple but has collision probability that grows as n/|S|. The fix is a uniqueness check + retry, which adds a DB roundtrip. The better default is a global atomic counter in Redis (INCR is single-threaded, atomic, eliminates collisions entirely) with Base62 encoding. The staff-level concern is the counter as a single point of failure: pre-allocated ID ranges per app server (each batch-fetches 1,000 IDs, eliminating per-request Redis coordination), and the counter node failing means temporary unavailability of writes but no data loss
>
> [!TIP]
> **🟢 Staff+** — Mention the predictability risk: sequential codes are enumerable. Mitigation: XOR the counter with a secret key before encoding, or accept that short URLs are meant to be shared publicly anyway
#### Deep dive 2: Fast redirects at scale (the read scaling problem)
> [!CAUTION]
> **🔴 Weak** — add a database index on short_code and query on every redirect
>
> [!WARNING]
> **🟡 Strong** — this is what makes Bitly hard — 100M DAU clicking links 10× per day = 11,600 read QPS, with viral links creating hot keys at 100× that rate. A database index on short_code is necessary but not sufficient. The right answer layers caching: Redis with cache-aside pattern absorbs 99% of reads
>
> [!TIP]
> **🟢 Staff+** — cache TTL should be set equal to or shorter than URL expiration so expired URLs auto-evict from cache (otherwise you serve expired redirects). For hot keys (viral links), add a local in-process LRU cache on each app server — zero network hops, handles traffic spikes that would otherwise hammer Redis. For global-scale: CDN caching of the 302 response itself with a short Cache-Control header removes Redis from the critical path entirely for the most popular links
#### Deep dive 3: Scaling to 1B URLs and 100M DAU (the DB and infrastructure problem)
> [!CAUTION]
> **🔴 Weak** — scale the database vertically and add read replicas
>
> [!WARNING]
> **🟡 Strong** — storage is not the hard part — 1B × 500 bytes = 500 GB, comfortably on a single PostgreSQL node with read replicas. The hard part is write coordination for the counter and horizontal scaling of stateless redirect servers. Senior answer: stateless redirect service behind a load balancer, Redis cluster for the counter, PG primary + read replicas
>
> [!TIP]
> **🟢 Staff+** — the counter service is the hidden bottleneck — pre-allocated ID ranges mean app servers can generate IDs locally without any network call (counter node becomes a periodic flush rather than a per-request bottleneck). Multi-region: deploy read replicas and Redis caches regionally, write counter remains in one region (acceptable because writes are rare). For URL expiration at scale: don't run a full table scan — use a time-indexed expiry column and batch-delete expired rows during low-traffic windows
_Why the deep dives connect to the scaling problem: The scaling pain is "reads explode while writes stay small." Deep dive 1 solves write uniqueness. Deep dive 2 solves read performance. Deep dive 3 solves infrastructure capacity. Name this arc explicitly in the interview — it shows architectural thinking, not just pattern recall._
Interview script
1. Requirements-first script.
2. "Before I design: a few quick clarifying questions. Are we building a public service like Bitly, or an internal tool? And do we care about analytics — click counts by geography, referrer — or just the redirect?"
3. "Great — public service, basic analytics out of scope for now. So my core features are: create a short URL from a long URL, support optional custom alias and expiration, and redirect via the short URL. I'll keep auth and abuse prevention out of scope unless you want them."
4. "For non-functionals: I'd assume 100M DAU, 1B total URLs, read-heavy — maybe 100:1 reads to writes. The main NFRs are fast redirects (sub-100ms), high availability, and globally unique short codes."
5. "API: POST /urls → {shortCode}. GET /{shortCode} → 302 redirect. That's the core contract."
6. "Data model: one table keyed by short_code — stores long_url, created_at, optional expiry, optional user_id. short_code is the primary key, long_url has an index for reverse lookup."
7. "Code generation: I'd use a global counter + Base62 encoding. Counter gives uniqueness guarantees without collision handling. Base62 gives 7 characters for 3.5 trillion possible codes — plenty. Tradeoff: codes are predictable, but that's fine for a public URL shortener."
8. "For read scaling: Redis cache with cache-aside. On redirect, check Redis first. Miss → hit PostgreSQL → populate Redis. 99% of reads served from cache. For viral links: local in-process LRU on each app server — eliminates Redis entirely for the hottest codes."
9. "302 over 301 — I want redirects to hit our server so we can track analytics if needed later. 301 caches permanently in the browser and we lose that."
Whiteboard
```
+-------------------+
| Clients |
| browser mobile app|
+---------+---------+
|
v
+-------------------+
| Load Balancer |
+----+---------+----+
| |
write path | | read path
v v
+----------------+ +----------------+
| Write Service | | Read Service |
+-------+--------+ +-------+--------+
| |
| |
v v
+----------------+ +----------------+
| Redis Counter | | Redis Cache |
| atomic INCR | | short -> long |
+-------+--------+ +-------+--------+
| |
| | cache miss
| v
| +--------------------+
+---------->| Postgres |
| short_code PK |
| long_url |
| expiration |
+---------+----------+
|
v
+--------------------+
| Background Cleanup |
| delete expired URLs|
+--------------------+
```
If you want to say it out loud, keep it simple. Clients hit a load balancer. Writes go to a write service, which gets a unique ID from Redis Counter, converts it to base62, and stores the mapping in Postgres. Reads go to a read service, which checks Redis Cache first, falls back to Postgres on a miss, checks expiration, and returns a 302 redirect.
If you want a slightly stronger version, you could add a CDN in front of the read path for hot links, but I would start with the sketch above in an interview.
The client splits a file into 4 MB chunks and computes a SHA-256 hash per chunk. It sends the hash list to the Metadata API, which checks which hashes are already stored. Only missing chunks are uploaded — directly to S3 via a pre-signed URL. The Metadata API writes the file-to-chunk mapping to PostgreSQL. Downloads flow through a CDN.
Problem
Dropbox solves cloud file storage and sync. Let users upload a file once, store it reliably, access it from any device, share it with others, and keep copies in sync.
Two layers: durable blob storage for the actual file bytes, and metadata + change tracking for ownership, sharing, and sync.
Failures
**S3 upload fails mid-way**
Large file partially uploaded. Client shows error. Resuming from scratch is expensive for GB-sized files.
_Fix:_ TUS or S3 multipart upload — track uploaded chunks, resume from last successful part. Store upload state in DB.
**Sync notification missed (device offline)**
Device misses a change event, shows stale file version.
_Fix:_ Don't rely on push alone. On reconnect, device sends its last-known state vector. Server diffs and sends all missed changes. Pull-on-reconnect as fallback to push.
**Metadata DB goes down**
Can't create, read, or share files. Uploads are blocked (can't get pre-signed URL without metadata record).
_Fix:_ PG primary + synchronous standby with automatic failover (Patroni). RTO < 30s. In-flight uploads continue directly to S3.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 50M DAU, avg 2 uploads/day at avg 5 MB, 100 downloads/day at avg 3 MB |
| Read QPS | 50M × 100 / 86400 ≈ 58K download QPS — nearly all served by CDN |
| Write QPS | 50M × 2 / 86400 ≈ 1,160 upload QPS — direct to S3 |
| Storage | 50M users × 15 GB avg free tier ≈ 750 PB — multi-region S3, lifecycle to cold tier after 90 days |
| Cache math | Metadata per file ≈ 200 bytes × 10B files = 2 TB metadata DB — needs sharding by user_id |
| Verdict | Storage cost dominates. Dedup is critical — even 20% dedup rate saves 150 PB. CDN absorbs download bandwidth. |
Design decisions
**Chunk size**
→ 4 MB chunks
Balances parallelism (multiple chunks in-flight), dedup granularity, and overhead. Smaller chunks = better dedup but more metadata. Larger = less overhead but worse dedup and resume granularity.
_Revisit when:_ Variable chunk sizing (content-defined chunking) for better dedup on large binary files.
**Direct S3 upload vs. proxying through app servers**
→ Direct S3 via pre-signed URL
Proxying 5 MB files through app servers at 1,160 QPS = ~5.8 GB/s through app tier. Completely unnecessary. Pre-signed URLs give the same security without the load.
_Revisit when:_ Never revisit — proxying is always wrong for large binary objects.
**Per-chunk dedup vs. per-file dedup**
→ Per-chunk (SHA-256)
Per-file dedup only helps for exact duplicates. Per-chunk helps when large files share common sections (e.g., only the last section of a document changed).
_Revisit when:_ Content-defined chunking (CDC) gives better dedup ratios for structured files.
Follow-up Q&A
**How do you handle conflicts when two devices edit the same file offline?**
Vector clocks or last-write-wins per chunk. On conflict: create a conflict copy (like Dropbox does), notify user to resolve. Don't silently overwrite — data loss is worse than a conflict notification.
**How do you handle a user with 1TB of files?**
Same architecture — chunking means we never load the whole file. The chunk manifest for a 1TB file is just a longer list of hashes. The expensive part is the initial upload bandwidth, not the server architecture.
**How would you implement sharing with permissions?**
SharedFiles table: (file_id, shared_with_user_id, permission_level, created_at). Check on every file operation. At scale, cache permission checks in Redis (short TTL, invalidate on permission change).
**What if the same file is uploaded by 1M users?**
SHA-256 dedup: stored once, shared metadata. Only first upload writes to S3. All subsequent uploads just create a new metadata row pointing to the same chunk hashes. True content-addressable storage.
**How do you handle versioning?**
Each save creates a new FileVersion row with a pointer to chunk manifest. Keep last N versions (configurable). Diff between versions = diff of chunk lists — show which chunks changed without re-downloading the file.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: success rate, latency, active users. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: minutes of read unavailability acceptable; rebuild cache from DB. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Direct upload to single S3 bucket. PG for metadata. No chunking, no dedup. Pre-signed URLs for access. Handles early adopters with small files.
**v2 — Scale** — Chunking + SHA-256 dedup. CDN for downloads. Sync notifications via WebSocket. Versioning. PG sharded by user_id. Handles millions of users.
**v3 — Optimize** — Content-defined chunking for better dedup. Cold storage tier for old chunks (S3 Glacier). Delta sync — send only changed bytes within a chunk. Background dedup job reconciles orphaned chunks.
Why it's hard to scale
The hard part is moving and syncing very large files cheaply and reliably. In Dropbox, the bottleneck is not just request count. It is the sheer amount of data, long upload times, and the fact that users expect the same file to appear quickly across devices.
A few things make this hard. Large files can time out, fail halfway, and need resume support. Downloads are heavy too, especially for users far from your storage region, so you need direct blob storage access and often a CDN. Sync is also tricky because you need to notice file changes, push updates to other devices, and recover if a device misses a notification. On top of that, sharing and permissions add metadata lookups, while reliability means you need durable storage and recovery if a server dies. So the core scaling pain is big blobs plus cross-device coordination, not just more API servers.
Key points
- **Metadata through your service** — Control plane: auth, metadata, sharing, URL signing. Heavy file bytes go around your service, not through it.
- **Blob storage** — Store actual file contents in S3. Never proxy large files through app servers.
- **Pre-signed URL upload** — Client uploads directly to blob storage. Bypasses your servers entirely.
- **CDN download** — Signed CDN URL for downloads. Low latency for users far away.
- **Chunking** — Split large files into 4 MB chunks on the client. Gives progress, retry, and resume.
- **SHA-256 dedup** — Same hash = same content = already stored. Global dedup across all users for free.
- **Delta sync** — On update, only upload changed chunks. This is what makes sync feel fast on large files.
- **Sharing** — Separate SharedFiles table — makes "files shared with me" queries fast.
> One interview sentence: Dropbox is a metadata service plus blob storage plus CDN, with direct upload, direct download, and a sync agent for change tracking.
Tradeoffs
**Upload through app vs direct to blob** — Direct is cheaper and faster. Tradeoff: more coordination around upload state and failures.
**Download from origin vs CDN** — CDN is better for low-latency global reads. Tradeoff: cost and cache invalidation.
**Sharelist in file vs separate table** — Separate table makes "files shared with me" fast. Tradeoff: another write path.
**Push sync vs polling** — Push gives near real-time updates but harder to run reliably. Polling is the fallback.
> "I chose the more scalable design for file transfer and reads, accepting extra complexity for better performance."
Deep dives
#### Deep dive 1: Large file uploads — chunking, resumability, and deduplication
_The scaling pain is moving and syncing large files cheaply and reliably_
> [!CAUTION]
> **🔴 Weak** — upload file to S3, done
>
> [!WARNING]
> **🟡 Strong** — client-side chunking (4 MB chunks) with SHA-256 fingerprint per chunk. Before uploading, client sends hash list to server — server responds with which hashes are new. Only missing chunks are transferred. This is content-addressable storage: same chunk stored once globally
>
> [!TIP]
> **🟢 Staff+** — chunk size is a design decision with real tradeoffs — smaller chunks give better dedup granularity and retry precision but more metadata overhead and round-trips; larger chunks have less overhead but worse dedup and waste bandwidth if a chunk fails mid-upload. Content-defined chunking (CDC using rolling hash, e.g., Rabin fingerprint) gives better dedup ratios for structured files (docs, code) by finding natural chunk boundaries rather than fixed offsets. The upload flow must be resumable: store upload state per (file_id, chunk_offset) so a failed upload resumes from the last committed chunk, not from scratch
#### Deep dive 2: Sync across devices — reliability and conflict resolution
_The scaling pain is cross-device coordination — detecting changes, pushing updates, and recovering from missed notifications_
> [!CAUTION]
> **🔴 Weak** — polling
>
> [!WARNING]
> **🟡 Strong** — WebSocket or SSE push for change notifications, with polling as a fallback for reconnect. The hard problem is what happens when two devices edit the same file while one is offline. Dropbox's actual approach: last-write-wins per file, conflicts result in a conflict copy (two files) rather than silent data loss
>
> [!TIP]
> **🟢 Staff+** — vector clocks or file version numbers allow the server to detect when a client's local state diverges from the server's state, enabling explicit conflict presentation rather than silent overwrites. The sync state machine per device: (synced → local_change_pending → uploading → synced) and (synced → remote_change_available → downloading → synced). On reconnect, client sends its last-known state vector; server diffs and returns the minimal change set
#### Deep dive 3: Security, sharing, and performance
> [!CAUTION]
> **🔴 Weak** — store the file URL in a shared link, serve directly from S3 with a public ACL
>
> [!WARNING]
> **🟡 Strong** — sharing model: SharedFiles table (file_id, shared_with_user_id, permission, created_at)
>
> [!TIP]
> **🟢 Staff+** — permission check on every file operation must be fast — cache permission results in Redis with short TTL, invalidate on any permission change. For large organizations: hierarchical permissions (team → folder → file) require careful data modeling to avoid O(n) permission lookups. Security: pre-signed S3 URLs with short expiry (15 min) for downloads — URL cannot be reused after expiry, prevents link sharing beyond the intended recipient. For compliance: server-side encryption (SSE-S3 or SSE-KMS), access logs for auditing, retention policies. Performance: CDN in front of S3 with signed cookies for authenticated users — reduces S3 egress costs and improves global latency
_Why the deep dives connect to the scaling problem: "Big blobs plus cross-device coordination." Deep dive 1 solves the blob problem (chunking + dedup). Deep dive 2 solves coordination (sync state machine + conflict handling). Deep dive 3 solves correctness and performance at scale._
Interview script
1. Use a simple script: upload, download, sharing, sync, then one deep dive.
2. "I'll design Dropbox as a cloud file storage and sync system. Core requirements: upload, download, sharing, and automatic sync. I'll prioritize availability over strict consistency."
3. "Key insight: metadata goes through my service, file bytes go around it. The File Service is the control plane — auth, permissions, metadata, and signed URLs."
4. "For uploads, the client asks the File Service for a presigned upload URL. Client uploads directly to blob storage. After upload, storage notifies the backend."
5. "For downloads, the service returns a signed CDN URL after auth checks. Client downloads from CDN, which fetches from blob storage on cache miss."
6. "For sync, each device runs a sync agent. Local changes trigger uploads. Remote changes come via WebSocket push with polling as fallback."
7. "The main tradeoff: sending files through the app server is simpler but direct upload scales much better."
Whiteboard
```
+----------------------+
| Desktop / Mobile |
| Web Client + Sync |
| Agent |
+----------+-----------+
|
auth, metadata APIs, change feed
|
v
+----------------------+
| LB / API Gateway |
+----------+-----------+
|
v
+----------------------+
| File Service |
| - authz checks |
| - file metadata |
| - share management |
| - presigned URLs |
| - signed CDN URLs |
+----+------------+----+
| |
metadata rw | | change events
v v
+----------------+ +------------------+
| FileMetadataDB | | Notification / |
| - files | | Change Service |
| - sharedFiles | | - WebSocket/SSE |
| - upload state | +--------+---------+
+----------------+ |
|
push updates |
v
+---------------+
| Client devices|
+---------------+
Upload path
-----------
Client -> File Service -> get presigned upload URL
Client -------------------------------> Blob Storage / S3
|
| upload complete event
v
File Service updates DB
Download path
-------------
Client -> File Service -> auth check + signed CDN URL
Client -------------------------------> CDN
|
cache miss|
v
Blob Storage / S3
Sharing path
------------
Client -> File Service -> update share records in DB
Sync path
---------
Local file change -> Sync Agent -> upload flow
Remote file change -> Notification Service pushes event
Missed event fallback -> Client polls `GET /files/changes?since=...`
```
The main mental model is this. Your app server is the control plane, not the data plane. It decides who can access a file and issues signed URLs, but the heavy file bytes go directly between the client, blob storage, and CDN.
If you want, I can also give you a more interview ready version that is smaller and faster to draw on a whiteboard.
Local delivery (GoPuff) Easy
Geospatial · Inventory consistency · Order flow
PostGISRedisPostgreSQLOptimistic lockGeohash
The user sends a location and item list. The Geo Service finds the nearest warehouse via geohash or PostGIS. For browsing, inventory is served from Redis — slight staleness is fine. At order placement the system uses a PostgreSQL transaction with row-level lock: decrement only if quantity > 0, otherwise reject.
Problem
Given a user's location, show which items are available for delivery within 1 hour, and let the user place an order without selling the same physical inventory twice.
The hard part: fast reads for availability, strongly consistent writes for orders.
Failures
**Inventory cache (Redis) goes stale during a spike**
Users see items as available that just sold out. They complete checkout, then get a cancellation. Poor UX.
_Fix:_ Short TTL (30s). On order placement, always recheck PG transactionally regardless of cache state. Cache is for browsing only, never for purchase commitment.
**PostgreSQL write node fails mid-order**
Order transaction rolls back. Inventory not decremented. Customer may be charged but order not created.
_Fix:_ 2-phase: charge authorization first (hold, not capture), then create order in PG, then capture charge. If PG fails, release auth. Idempotency key prevents double-charge on retry.
**Geo service slow on radius queries**
Availability check exceeds 100ms SLA. Bad mobile UX.
_Fix:_ Pre-compute geohash cells for all warehouses. Radius query becomes a set lookup on geohash prefixes. Cache warehouse list per geohash (changes slowly).
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 10M DAU, 5 availability checks/session, 1 order/3 sessions, avg 5 items/order |
| Read QPS | 10M × 5 / 86400 ≈ 578 availability QPS — dominated by Redis reads |
| Write QPS | 10M / 3 / 86400 ≈ 39 order QPS — low, but each touches 5 inventory rows atomically |
| Storage | 100K SKUs × 100 warehouses × 50 bytes ≈ 500 MB inventory table — tiny, fits in Redis entirely |
| Cache math | Full inventory in Redis ≈ 500 MB. Trivial. TTL 30s means at worst 30s staleness on availability. |
| Verdict | Writes are the hard part, not reads. 39 QPS of atomic inventory transactions — single PG node is fine up to ~500 QPS of orders. |
Design decisions
**Optimistic vs. pessimistic locking for inventory**
→ Pessimistic (SELECT FOR UPDATE)
Inventory contention is high for popular items during peak hours. Optimistic locking would cause high retry rates and bad UX. Pessimistic lock duration is <50ms so throughput is acceptable.
_Revisit when:_ Optimistic locking if order volume exceeds 5K QPS per warehouse (unlikely).
**Separate read and write paths vs. single DB**
→ Redis for reads, PG for writes
Availability checks are 10× more frequent than orders. Serving availability from PG would add unnecessary load. Redis with short TTL is the right tradeoff.
_Revisit when:_ Single DB path for simpler consistency if scale stays small.
**Geo service vs. in-DB geospatial query**
→ Separate geo service with geohash
PostGIS radius queries are powerful but slow at high QPS. Geohash prefix lookup is a simple index scan — much faster.
_Revisit when:_ PostGIS is fine up to ~1K QPS geo queries. Only separate at higher scale.
Follow-up Q&A
**How do you prevent overselling the last unit?**
SELECT FOR UPDATE on the inventory row inside the order transaction. Decrement only if quantity > 0, else roll back and return 'sold out'. Atomicity in PostgreSQL guarantees no two transactions decrement simultaneously.
**What happens if a user adds to cart but doesn't check out?**
Don't reserve inventory at cart time — reserve only at checkout. Cart is a soft state in Redis with TTL. Inventory is only committed when the order transaction commits.
**How do you handle a warehouse going offline?**
Health check pings warehouses every 30s. On failure: mark inactive in geo service, remove from availability results. Orders route to next nearest warehouse. Alert ops.
**How do you handle a sudden 10× spike in a neighborhood (e.g., bad weather)?**
Read path (Redis) handles spikes trivially. Write path (PG transactions) is the bottleneck. Queue order submissions with SQS. Workers drain queue at PG's max safe write rate. Show estimated wait time to user.
**How would you add real-time ETA?**
Separate routing service calls Google Maps / internal routing engine. ETA is an estimate, not a commitment. Cache ETA per (warehouse, delivery_zone) pair with 5-minute TTL. Driver tracking is a separate real-time system.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: success rate, latency, active users. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: minutes of read unavailability acceptable; rebuild cache from DB. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Single PG for inventory + orders. No geo service — just distance formula in app code. No caching. Works for 1-2 warehouses.
**v2 — Scale reads** — Redis for inventory browsing. Geo service with geohash. PG read replica for reporting. Handles 10+ warehouses, 100K DAU.
**v3 — Optimize** — Order queue with SQS for spike absorption. Predictive inventory replenishment from ML model. Real-time driver tracking as separate service. Dynamic delivery window pricing.
Why it's hard to scale
The hard part is that reads are huge, but writes must be correct. In this Gopuff style system, availability checks happen constantly and need to stay under 100ms, while actual orders are less frequent but need strong consistency so you never sell the same item twice.
So there are really two scaling pain points. First, availability is expensive because you are not just reading one row. You have to find nearby distribution centers, read inventory from several of them, and union the results fast enough for search-like traffic. Second, ordering creates contention because many users may try to buy the last unit at the same time, so inventory updates need atomic transactions or locking. A good mental model is that reads are broad and frequent, while writes are rare but delicate.
Key points
- **Two-path model** — Fast reads for availability. Safe writes for orders. Different consistency requirements.
- **Availability read** — Find nearby DCs via geohash, aggregate inventory, cache in Redis with short TTL.
- **Order write** — Recheck inventory and create order in one PostgreSQL serializable transaction.
- **Nearby service** — Coarse geographic filter first, then travel time estimation on the small candidate set.
- **Cache invalidation** — Expire affected Redis keys after an order decrements inventory.
> Open with the two-path split: browsing is eventually consistent (Redis), ordering is strongly consistent (PG row lock). Don't conflate them.
Tradeoffs
**Cache + replicas vs single DB for reads** — Cache + replicas is better because reads are huge and can tolerate slight staleness. Tradeoff: freshness vs speed.
**Single PG transaction vs distributed lock** — Single transaction is simpler and correct. Tradeoff: couples orders and inventory, bottleneck sooner.
**Distance filter vs travel time** — Distance filter is fast and cheap. Travel time is accurate but adds latency.
> For availability I'm trading read freshness for speed — Redis cache with 30s TTL. For ordering I'm choosing correctness with SELECT FOR UPDATE. Two paths, two consistency models, both intentional.
Deep dives
#### Deep dive 1: Availability reads at scale — the geo + inventory problem
> [!CAUTION]
> **🔴 Weak** — query PostgreSQL with PostGIS radius filter on every availability request
>
> [!WARNING]
> **🟡 Strong** — the scaling pain is that availability reads are expensive: find nearby DCs, read inventory from each, union results, respond in <100ms. This is not a simple DB read — it's a geospatial + multi-source aggregation under search-like QPS. Weak answer: query PG with PostGIS radius filter. Strong answer: pre-compute a geohash index of all DCs, cache inventory in Redis with short TTL (30s). Availability query = geohash prefix lookup (O(1)) + Redis HGETALL for each DC in range
>
> [!TIP]
> **🟢 Staff+** — inventory in Redis is a projection, not the source of truth — it gets stale within the TTL window. This is intentional: users browsing see approximately-fresh inventory, users purchasing get a real-time DB check. The two paths have different consistency requirements and must be explicitly separated. At high QPS, even Redis can become a bottleneck — read-through cache with local in-process LRU for the hottest items (fast movers, popular items) adds another layer
#### Deep dive 2: Order placement — preventing oversell under concurrent writes
_The core hard problem: many users buying the last unit simultaneously_
> [!CAUTION]
> **🔴 Weak** — check inventory then decrement (two operations — TOCTOU race)
>
> [!WARNING]
> **🟡 Strong** — SELECT FOR UPDATE within a PostgreSQL transaction — atomic lock, check, and decrement
>
> [!TIP]
> **🟢 Staff+** — tradeoff: pessimistic locking works but creates serialization under high concurrent orders for the same item. Optimistic locking (CAS via version number) is better under low contention, worse under high contention. For a local delivery system where popular items genuinely spike (weather events, flash sales), pessimistic is the safer default. SQS queue for order submission acts as a shock absorber: workers drain the queue at a safe DB write rate, and user sees "order processing" rather than an error. Idempotency key on every order prevents double-charge on client retry
#### Deep dive 3: Geo-routing — nearest DC with ETA, not just distance
> [!CAUTION]
> **🔴 Weak** — find the nearest DC by Euclidean distance
>
> [!WARNING]
> **🟡 Strong** — a production system doesn't just find the nearest DC by Euclidean distance — it finds the nearest DC that can fulfill the order within the delivery window. This requires: (1) drive time estimate from DC to delivery address (routing API, not just radius), (2) DC capacity check (is there a driver available?), (3) inventory check (does this DC have the items?)
>
> [!TIP]
> **🟢 Staff+** — this is a constraint satisfaction problem across three dimensions. The production approach is coarse + fine: geohash radius filter first (fast, eliminates 99% of DCs), then routing API call on the small candidate set (expensive, accurate). Cache routing estimates per (DC_geohash, delivery_zone) pair with 5-minute TTL to reduce routing API costs
_Why the deep dives connect to the scaling problem: "Reads are broad and frequent; writes are rare but delicate." Deep dive 1 solves the read problem. Deep dive 2 solves the write correctness problem. Deep dive 3 solves the geo-routing precision problem._
Interview script
1. Two-path script.
2. "Clarifying questions: are we building a dark-store model — our own warehouses — or a marketplace model routing to third-party stores? And what's the delivery promise — 30 minutes, on-demand?"
3. "Good — own warehouses, 30-min delivery. Core features: browse available items by location, place order, track delivery. Out of scope: driver routing optimization, demand forecasting, warehouse management."
4. "Two fundamentally different consistency requirements I'd name upfront: browsing inventory can be eventually consistent — a 30-second stale read is fine. Placing an order must be strongly consistent — two customers cannot buy the last unit."
5. "Browse path: Redis cache per warehouse, 30-second TTL. Geo service finds nearest warehouses via geohash prefix lookup. Client gets near-real-time inventory without hitting the DB."
6. "Order path: PostgreSQL SELECT FOR UPDATE inside a transaction. Lock the row, check quantity > 0, decrement. If quantity is 0: rollback, return sold-out. Atomic. No oversell possible."
7. "SQS queue in front of order processing acts as a shock absorber for spikes — bad weather, local events. Workers drain the queue at a safe DB write rate. Users see a short wait rather than an error."
8. "Key tradeoff: Redis is eventually consistent — a user might see an item as available in browse mode that sold out 29 seconds ago. That's acceptable. The purchase path always gets a fresh DB check regardless of cache state."
Whiteboard
```
+------------------+
| Client |
| Web or Mobile |
+---------+--------+
|
Availability API | Order API
|
+----------------+----------------+
| |
v v
+-----------------------+ +-----------------------+
| Availability Service | | Orders Service |
| read path | | write path |
+----------+------------+ +-----------+-----------+
| |
| asks for serviceable DCs | asks for serviceable DCs
v v
+---------------------------+
| Nearby Service |
| find DCs within 1 hour |
+-------------+-------------+
|
| candidate DCs
v
+---------------------------+
| Travel Time Service |
| external ETA estimation |
+---------------------------+
Availability read flow
----------------------
|
v
+-----------------------+
| Redis Cache |
| availability results |
| short TTL |
+----------+------------+
|
cache miss
v
+-----------------------+
| Postgres Read Replica |
| inventory reads |
+----------+------------+
|
v
+-----------------------+
| Partitioned by Region |
| inventory + items |
+-----------------------+
Order write flow
----------------
|
v
+-----------------------+
| Postgres Leader |
| serializable txn |
+----------+------------+
|
v
+-----------------------+
| Tables |
| Inventory |
| Items |
| Orders |
| OrderItems |
+-----------------------+
|
v
+-----------------------+
| Cache Invalidation |
| expire affected keys |
+-----------------------+
```
The mental model is simple. Availability is a fast read path that can tolerate slight staleness, so it uses Nearby Service, cache, and read replicas. Orders are the strict write path, so they go to the Postgres leader in one transaction so you do not double sell inventory.
If you were drawing this in an interview, I would show just these boxes first. Then I would say reads go through cache and replicas, while writes go through the leader with an atomic transaction.
RSS crawlers feed articles into Kafka. A dedup service computes a SimHash fingerprint per article — Hamming distance Elasticsearch powers search. Cassandra stores raw articles. Regional Redis sorted sets serve the feed path.
Problem
Organizing news from thousands of publishers into one fast scrollable feed. The system collects articles, stores metadata, and redirects users to the publisher site on click.
The challenge: fast aggregation + deduplication at scale, not hosting full articles.
Failures
**Redis feed cache evicts an entry during peak read**
Cache miss causes DB query — at 100M users × breaking news = millions of concurrent misses (thundering herd).
_Fix:_ Cache warming on publish: when new articles are added to a feed, proactively write to Redis rather than waiting for reads. Add probabilistic early expiration to prevent synchronized expiry.
**Crawler is blocked by a publisher**
Publisher's articles stop appearing. Users notice freshness degradation for that source.
_Fix:_ Exponential backoff on crawler errors. Multiple crawl strategies (RSS, scrape, webhook). Monitor per-source freshness SLA with alerts.
**Elasticsearch index falls behind during a breaking news event**
Search results don't show the most recent articles for a rapidly evolving story.
_Fix:_ Dedicated high-priority Kafka topic for breaking news. Separate fast-lane indexing pipeline with lower batch size. Monitor ES indexing lag with alerting at >30s.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 100M DAU, 10 feed refreshes/day, 1M articles ingested/day from 10K publishers |
| Read QPS | 100M × 10 / 86400 ≈ 11,600 feed read QPS — served from Redis |
| Write QPS | 1M articles / 86400 ≈ 12 ingest QPS — trivially small |
| Storage | 1M articles/day × 2 KB × 365 days × 3 years ≈ 2.2 TB article store in Cassandra |
| Cache math | 100M users × 50 article IDs × 8 bytes ≈ 40 GB Redis — fits on a beefy Redis cluster |
| Verdict | Read-dominant by 1000:1. Redis feed cache is the critical path. Ingestion is easy — only 12 QPS. |
Design decisions
**Pre-computed feeds vs. on-demand assembly**
→ Pre-computed regional Redis sorted sets
100M users × 11,600 QPS — on-demand assembly would require querying Cassandra per user per request. Impossible. Pre-compute on write (ingest), serve from Redis on read.
_Revisit when:_ On-demand assembly with a fast query cache if personalization requirements grow beyond regional buckets.
**Cursor pagination vs. offset**
→ Cursor (published_at + article_id composite)
Offset pagination is broken for feeds — new articles shift positions while the user scrolls, causing duplicates and gaps. Cursor is stateless and stable.
_Revisit when:_ Never use offset for real-time feeds.
**SimHash dedup vs. exact hash**
→ SimHash with Hamming distance threshold
Exact hash only catches identical articles. Same story reported by 50 publishers with slight wording differences = 50 duplicate articles. SimHash catches near-duplicates.
_Revisit when:_ More sophisticated NLP clustering (topic modeling) if higher quality story grouping is required.
Follow-up Q&A
**How do you handle a breaking news event with 100× normal traffic?**
Redis feed reads scale horizontally — add read replicas. The bottleneck is cache population during the event. Prioritize updating breaking-news topic feeds first. CDN-cache the feed API responses for 5s to absorb the spike tail.
**How do you personalize feeds without blowing up Redis storage?**
Cluster users into interest profiles (topic affinity vectors). Pre-compute one feed per cluster (thousands, not millions). On read, take the cluster feed and apply lightweight user-specific boosts in app layer.
**How fresh does the feed need to be?**
Explicitly define the SLA: breaking news < 2 min, general news < 10 min. Different ingestion pipelines serve different SLAs. Webhooks from publishers for < 2 min; polling for the rest.
**How do you prevent one spammy publisher from polluting feeds?**
Publisher trust score (authority signal). Rate limit new publishers. Editorial review queue for sources below a trust threshold. SimHash dedup also helps by collapsing near-duplicate spam.
**What's the hardest part to get right in production?**
Cache consistency during feed updates. When you update a regional feed, you have to atomically swap it — partial updates show users an inconsistent feed mid-scroll. Use Redis MULTI/EXEC or REPLACE rather than incremental append.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: success rate, latency, active users. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: minutes of read unavailability acceptable; rebuild cache from DB. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Poller fetches RSS feeds every 5 min. PG stores articles. Feed built on-demand with SQL. Simple and works for <100K users.
**v2 — Scale** — Kafka ingestion pipeline. Cassandra for article storage. Redis pre-computed regional feeds. SimHash dedup. Cursor pagination. Elasticsearch for search.
**v3 — Personalize** — User interest clustering. Personalized re-ranking in app layer. ML-based story clustering. Freshness SLA monitoring per source. Publisher trust scoring.
Why it's hard to scale
The hard part is that both sides scale at once. You are ingesting articles from thousands of publishers, while also serving a huge read-heavy feed to millions of users with very fresh content.
There are three main pain points. First, feed reads are massive, especially during breaking news, so you cannot query the database for every request and still stay under 200ms. Second, the feed keeps changing while users scroll, which makes simple page-number pagination cause duplicates or missed articles. Third, freshness matters a lot, so ingestion is not just batch ETL work. You need to discover new articles quickly, update regional feeds fast, and keep caches fresh without overwhelming publishers or your own systems.
Key points
- **Two pipelines** — Write pipeline ingests from publishers. Read pipeline serves cached feeds to users.
- **Collect** — RSS, publisher APIs, or web scraping.
- **Dedup** — SimHash fingerprint per article. Hamming distance < 3 = near-duplicate, discard.
- **Precompute feeds** — Regional Redis sorted sets. New articles update them async. Reads are fast cache lookups.
- **Pagination** — Cursor-based, not page numbers. Offset pagination causes gaps when new articles arrive.
- **Thumbnails** — Copy to object storage + CDN rather than hotlinking publisher images.
> Lead with the read/write ratio: ingest is trivial (12 QPS), serving is massive (11,600 QPS). Everything flows from that asymmetry — precompute on write, serve from cache.
Tradeoffs
**Offset pagination vs cursor** — Offset causes gaps when new articles arrive. Cursor is correct for infinite scroll.
**DB reads vs Redis precomputed feeds** — DB reads are simpler. Redis feeds are much faster for 100M users but add update logic.
**Polling vs webhooks for ingestion** — Polling works without coordination, content arrives later. Webhooks are fresher but need publisher adoption.
> I favor precomputed regional feeds over on-demand assembly, cursor pagination over offsets, and SimHash near-dedup over exact dedup. Each tradeoff buys read speed at the cost of some write complexity.
Deep dives
#### Deep dive 1: Feed generation — precomputed vs. on-demand at 100M DAU
> [!CAUTION]
> **🔴 Weak** — query Cassandra per request, assemble feed on-demand
>
> [!WARNING]
> **🟡 Strong** — the scaling pain is that you can't query the database for every feed request at 11,600 QPS and stay under 200ms. Weak answer: query Cassandra per request. Strong answer: pre-compute regional Redis sorted sets (score = publish timestamp) and serve feeds from cache. The hard part is keeping pre-computed feeds fresh during breaking news when articles arrive at high frequency
>
> [!TIP]
> **🟢 Staff+** — write-through feed population — when an article passes dedup and is stored, the ingestion service immediately pushes its ID into the relevant regional sorted sets. This inverts the cache pattern: writes are the fan-out, reads are O(1) ZREVRANGE. Tradeoffs to name: (1) feed storage cost (100M users × 50 IDs × 8 bytes = 40 GB Redis — clusters needed), (2) feed staleness during cache update, (3) thundering herd if a regional cache expires simultaneously. Fix the last one with probabilistic early expiration
#### Deep dive 2: Cursor-based pagination — why it's mandatory for live feeds
_Offset pagination is broken for live feeds: new articles shift existing positions while the user scrolls, causing duplicates (article appears twice) or gaps (article skipped)_
> [!CAUTION]
> **🔴 Weak** — use offset pagination, acknowledge the issue
>
> [!WARNING]
> **🟡 Strong** — cursor pagination using a composite cursor (published_at + article_id). The cursor encodes where the user is in the feed at the moment they fetched the previous page — new articles added to the top don't shift relative positions below the cursor
>
> [!TIP]
> **🟢 Staff+** — cursor must be stable under concurrent feed updates. Using (published_at, article_id) as cursor is stable because we sort by published_at DESC, article_id DESC — inserting new articles at the top doesn't reorder articles below any existing cursor position. Implementation: SELECT * FROM feed WHERE (published_at, article_id) < (cursor_time, cursor_id) ORDER BY published_at DESC, article_id DESC LIMIT 20
#### Deep dive 3: Article deduplication and story clustering
> [!CAUTION]
> **🔴 Weak** — use exact SHA-256 hashing — only catches identical articles
>
> [!WARNING]
> **🟡 Strong** — SimHash dedup: each article gets a fingerprint by tokenizing text → compute tf-idf weights → hash weighted term vector to 64-bit integer. Hamming distance < 3 between two fingerprints = near-duplicate. This catches the same story reported by 50 publishers with slight wording differences
>
> [!TIP]
> **🟢 Staff+** — SimHash lookup requires comparing a new fingerprint against all stored fingerprints — at 1M articles/day this is O(N) per insert. The solution: partition fingerprints by their first K bits (locality-sensitive hashing) so only fingerprints in the same bucket need comparison. Story clustering goes further: group near-duplicates into a "story" entity with the highest-authority source as the canonical article. Ranking: recency × source authority score (PageRank-like, precomputed per domain)
_Why the deep dives connect to the scaling problem: "Both sides scale at once — ingestion and reads." Deep dive 1 solves read scaling. Deep dive 2 solves pagination correctness. Deep dive 3 solves the content quality problem that makes the system valuable._
Interview script
1. Asymmetry-first script.
2. "Clarifying questions: are we aggregating from a fixed set of publisher RSS feeds, or also crawling arbitrary web pages? And does personalization matter — per-user feeds — or just regional topic feeds?"
3. "Good — RSS plus crawl, regional feeds initially with per-user personalization later. Core features: ingest articles, deduplicate, rank, serve feed. Out of scope: publisher partnerships, comment systems."
4. "The key asymmetry to name immediately: ingest is trivial — ~12 writes/sec. Serving is the hard problem — 11,600 feed read QPS. Everything should optimize for reads."
5. "Ingest pipeline: crawler fetches RSS feeds, extracts article text, pushes to Kafka. SimHash fingerprinting detects near-duplicates (Hamming distance < 3). Unique articles stored in Cassandra, indexed in Elasticsearch."
6. "Feed generation: pre-compute regional Redis sorted sets on write. When a new article lands, push its ID into the relevant topic and region sets immediately. Feed read = ZREVRANGE — sub-millisecond, no query needed."
7. "Cursor pagination over offsets: new articles shift offset-based positions while the user scrolls, causing duplicates and gaps. Cursor using (published_at, article_id) is stable under concurrent inserts."
8. "Freshness: for breaking news, the ingest-to-index pipeline needs to complete in under 2 minutes. Monitor indexing lag per source. Webhook-based ingest from major publishers for the fastest path."
Whiteboard
```
+----------------------+
| News Publishers |
| RSS APIs Webhooks |
+----------+-----------+
|
v
+-------------------------------+
| Data Collection Service |
| poll feeds parse ingest media |
+---------------+---------------+
|
+------------------+------------------+
| |
v v
+----------------------+ +----------------------+
| Article Database | | Object Storage |
| articles publishers | | thumbnails images |
+----------+-----------+ +----------+-----------+
| |
| new article writes |
v v
+----------------------+ +----------------------+
| CDC Event Stream | | CDN |
| change notifications | | serve thumbnails |
+----------+-----------+ +----------------------+
|
v
+------------------------------+
| Feed Generation Workers |
| update regional feed caches |
+--------------+---------------+
|
v
+------------------------------+
| Redis Feed Cache |
| feed:US feed:UK sorted sets |
| recent article ids by time |
+--------------+---------------+
|
v
+-----------+ +----------------------+ +------------------+
| Client |-->| API Gateway |-->| Feed Service |
| web mobile| | auth rate limit | | get feed paginate|
+-----------+ +----------------------+ +---------+--------+
|
+--------------------+-------------------+
| |
v v
+----------------------+ +----------------------+
| Redis Feed Cache | | Article Database |
| primary read path | | cache miss fallback |
+----------------------+ +----------------------+
|
v
+------------------+
| Feed Response |
| title summary |
| thumbnail url |
| publisher url |
+------------------+
```
The main idea is two pipelines. One pipeline ingests articles from publishers and stores article metadata plus thumbnails. The other pipeline serves users by reading precomputed regional feeds from Redis so feed requests stay fast.
If you want the best interview version, draw the high level boxes first, then call out one improvement. Use cursor pagination for infinite scroll and Redis precomputed regional feeds for low latency. That shows the core system clearly without overcrowding the board.
Phase 1: SETNX seatId in Redis with a 10-minute TTL — atomic soft lock, first caller wins. Phase 2: payment success → hard-commit SOLD to PostgreSQL. TTL expiry auto-releases stuck holds. Elasticsearch handles event search so reads never hit the primary DB.
Problem
Selling seats for a live event without double-booking, while millions of people browse, search, and buy simultaneously.
The hard part: high availability for browsing but strong consistency for booking — one seat can only be sold once.
Failures
**Redis seat hold TTL expires while user is filling in payment info**
User completes checkout, gets 'seat no longer available' error. Terrible UX on a high-anxiety purchase.
_Fix:_ Extend TTL on active checkout sessions (heartbeat from client every 2 min). Give generous initial TTL (15 min). Warn user at 2 min remaining.
**Payment service is slow / down during onsale**
Users hold seats via Redis but can't complete payment. TTLs expire, seats released — perceived as site failure.
_Fix:_ Pre-authorize payment before Redis hold (faster path). Async payment confirmation. Queue completed payments if PSP is slow.
**Elasticsearch falls behind during onsale traffic spike**
Search for events is slow/stale. Users can't find events to buy.
_Fix:_ ES cluster pre-scaled before announced onsale. Read-only event data mostly cacheable. CDN cache event detail pages (not seat maps).
Estimation
| Field | Value |
|-------|-------|
| Assumptions | Taylor Swift onsale: 2M users hit 'buy' within 60s. 50K seats. 10M seat map refreshes in first hour. |
| Read QPS | 10M seat map reads / 3600s ≈ 2,778 read QPS — needs aggressive caching |
| Write QPS | 2M booking attempts / 60s ≈ 33,333 booking QPS — this is the spike problem |
| Storage | 50K seats × 200 bytes = 10 MB per event — trivially small, entirely in Redis |
| Cache math | Seat map cached with 5s TTL means reads = 1 DB read / 5s, serving 2,778 QPS from cache. Without cache: 2,778 QPS to PG = death. |
| Verdict | The 33K booking QPS spike is the actual problem. Mitigate with virtual queue — let only 10K users into booking flow at once. |
Design decisions
**Redis TTL hold vs. DB-level lock**
→ Redis SETNX with TTL
DB-level locks held for 10 minutes across 50K seats during onsale would create massive lock contention. Redis SETNX is O(1), distributed, auto-expiring. PG is only touched for final commit.
_Revisit when:_ DB-level advisory locks acceptable for small events (<5K seats). Redis overkill at that scale.
**Virtual queue vs. open access**
→ Virtual waiting room
Uncapped traffic at 33K booking QPS would require 33× normal infrastructure capacity for a 60-second spike. Queue smooths demand. Users prefer 'you are #14,532 in queue' to a 500 error.
_Revisit when:_ Open access fine for events with low demand-to-supply ratio.
**Per-seat vs. per-section holds**
→ Per-seat Redis keys
Users select specific seats. Section-level holds would require complex seat-within-section allocation logic. Per-seat is simpler and maps directly to inventory.
_Revisit when:_ Section-level + best-available algorithm for mobile-first products where users don't care about specific seat.
Follow-up Q&A
**Two users select the same seat simultaneously — what happens?**
SETNX is atomic. First request sets the key and succeeds. Second request finds key already set and returns 'seat taken'. No race condition possible — Redis single-threaded command execution guarantees this.
**How do you handle bots buying all tickets instantly?**
CAPTCHA at queue entry. Rate limiting per IP and per account. Browser fingerprinting. Verified fan presale (account age/purchase history required). Throttle accounts with no prior purchase history.
**What if Redis goes down during an onsale?**
This is catastrophic. Mitigate: Redis Sentinel or Cluster for HA. If Redis fails, fall back to PG-level advisory locks — slower but correct. Pre-onsale: verify Redis health explicitly, not just assume.
**How do you handle seat map updates (seat released after TTL)?**
On TTL expiry, Redis keyspace notification triggers an event. Seat map cache is invalidated. Next seat map request rebuilds from PG + active Redis holds. SSE pushes updated seat map to active clients.
**How would you scale to 10 simultaneous onsales?**
Each event is sharded independently — different Redis keyspace, different booking service instances. Horizontal scaling by event_id. The virtual queue for each event is independent.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: throughput, queue lag, cache effectiveness. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: async replication lag <30s; failover promotes read replica. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — PG for all seats. Optimistic locking with retries. Works for small events (<1K seats, low contention). Simple but breaks under any real onsale load.
**v2 — Handle onsales** — Redis SETNX per seat. PG only for final commit. Virtual waiting room for popular events. SSE for real-time seat map. Handles major onsales.
**v3 — At scale** — Pre-scale infrastructure for announced onsales. Regional deployment for global artists. ML-based demand prediction for queue sizing. Dynamic pricing tier support.
Why it's hard to scale
The hard part is contention under huge spikes. Ticketmaster is not just a read heavy system. It is a system where millions of people may fight over the exact same tiny set of seats at the same moment.
That creates three main scaling pain points. First, the event page and seat map get hammered by refresh traffic, so reads spike hard and cached data goes stale fast. Second, booking is a hotspot write problem because many users try to reserve the same seat, so you need very careful coordination to avoid double booking. Third, search and queueing have to stay responsive during the surge, or users will overload the system before they even reach checkout. A good mental model is broad traffic everywhere, but extreme contention at a few hot seats.
Key points
- **3 core flows** — View events, search events, book tickets.
- **Core tension** — Browsing wants high availability. Booking wants strong consistency.
- **Booking pattern** — Reserve seat (Redis TTL hold) → user pays → confirm (PG commit). Never hold a DB transaction open for minutes.
- **Scale reads** — Cache event details. Elasticsearch for search, not SQL LIKE scans.
- **Peak demand** — SSE for real-time seat map updates. Virtual waiting room for extreme onsales.
> Lead with the core tension: browsing wants high availability, booking wants strong consistency. Name this split in your first sentence — it frames every decision that follows.
Tradeoffs
**Redis seat holds vs DB-only holds** — Redis gives better UX and auto-expiry. Tradeoff: reserved seats live outside the DB so the seat map must reflect Redis state.
**Elasticsearch vs SQL for search** — ES gives fast fuzzy full-text search. Tradeoff: sync complexity — index can lag behind PostgreSQL.
**Virtual waiting room vs no queue** — Waiting room protects the system and improves fairness. Tradeoff: user friction.
> I optimize reads aggressively (cache, CDN, Elasticsearch) but keep the final purchase path strongly consistent (Redis SETNX hold → PG commit). Never weaken the booking path for the sake of throughput.
Deep dives
#### Deep dive 1: Seat reservation — preventing double booking under extreme contention
_This is the defining hard problem for Ticketmaster. The scenario: Taylor Swift onsale, 2M users, 50K seats, all trying to book in 60 seconds_
> [!CAUTION]
> **🔴 Weak** — database SELECT + UPDATE
>
> [!WARNING]
> **🟡 Strong** — Redis SETNX per seat_id with 10-minute TTL creates a soft hold atomically (first caller wins, Redis is single-threaded so no race). On payment success, a short PG transaction commits the hard booking
>
> [!TIP]
> **🟢 Staff+** — the two-phase design explicitly separates hold duration (user has time to pay) from commit duration (DB transaction is < 100ms). Never hold a DB transaction open for 10 minutes — lock contention at scale would serialize all booking requests. TTL auto-release eliminates abandoned carts without any cleanup job. Key failure mode to proactively surface: Redis hold state and PG booking state can diverge if a crash occurs between them — on recovery, reconcile by querying PG for completed payments and releasing any Redis holds for unpaid seats
#### Deep dive 2: Seat map real-time updates — SSE fan-out under spike traffic
_During an onsale, users need to see seat availability update in near-real-time as others hold and release seats_
> [!CAUTION]
> **🔴 Weak** — HTTP polling every 5 seconds
>
> [!WARNING]
> **🟡 Strong** — SSE for server-push updates, Kafka for event distribution, delivery servers partitioned by event_id
>
> [!TIP]
> **🟢 Staff+** — SSE fan-out math matters — 2M users watching the same event × 1 update/hold event = massive write amplification. Mitigation: (1) coalescing — batch seat status changes into 500ms update windows rather than per-event pushes; (2) broadcast the full seat map diff rather than individual seat changes; (3) for extreme events, short TTL CDN caching of the seat map image reduces real-time update pressure. The seat map itself is event-specific static data — cache aggressively. Only availability state is dynamic
#### Deep dive 3: Virtual waiting room — protecting the system under extreme spikes
_Without a queue, 2M simultaneous users hit booking flow in 60 seconds = 33K booking QPS_
> [!CAUTION]
> **🔴 Weak** — scale horizontally
>
> [!WARNING]
> **🟡 Strong** — virtual waiting room caps entry into booking flow at a rate the system can handle (e.g., 5K users/minute). Ticket purchasing is a funnel: users in waiting room → hold seat → payment → confirmation. The queue smooths the spike
>
> [!TIP]
> **🟢 Staff+** — design: waiting room assigns users a randomized position (not FIFO — prevents queue jumping bots who connect milliseconds early). Position is stored in Redis sorted set with score = random. Users poll their position. When it's their turn, they receive a signed token that grants entry to booking flow (token has 5-minute TTL). The token prevents users from sharing their queue position. Estimate: with 2M users and a 5K/min entry rate, median wait ≈ 200 minutes. For popular events this is expected and communicated to users upfront
_Why the deep dives connect to the scaling problem: "Extreme contention at a few hot seats." Deep dive 1 solves booking correctness. Deep dive 2 solves real-time UX. Deep dive 3 solves traffic smoothing._
Interview script
1. Requirements-first, tension-first script.
2. "Quick clarifications: are we handling the full flow — event browsing, search, and ticket purchase? And what's the peak demand scenario — a major onsale like Taylor Swift?"
3. "Got it. Core features: view events, search events, book tickets. The defining constraint: browsing should be highly available and low-latency. Booking must be strongly consistent — we can never double-sell a seat."
4. "Scale: I'll assume 10M normal DAU, with onsales hitting millions of concurrent users for a single event in under 60 seconds. That's the hard case to design for."
5. "High-level: three services behind an API gateway — Event Service, Search Service, Booking Service. PostgreSQL is source of truth. Redis for seat holds. Elasticsearch for full-text event search."
6. "Booking deep-dive — this is where it gets interesting. I'd use a two-phase approach. Phase 1: Redis SETNX with a 10-minute TTL per seat_id. SETNX is atomic — first caller wins, no race condition possible. Phase 2: on payment success, a short PostgreSQL transaction marks the ticket SOLD and releases the Redis hold."
7. "For onsales: a virtual waiting room. Cap entry into the booking flow at a rate the system can handle — say 5K users per minute. Users in the queue get a position and a signed token that grants them entry. This converts a 33K QPS spike into a steady 5K QPS stream."
8. "Key tradeoff I want to name explicitly: Redis holds and PostgreSQL records can diverge if a crash occurs between them. On recovery, reconcile by querying PostgreSQL for completed payments and releasing any orphaned Redis holds."
Whiteboard
```
+-------------------+
| Clients |
| Web / Mobile |
+---------+---------+
|
HTTPS|
v
+-------------------+
| Load Balancer |
+---------+---------+
|
v
+-------------------+
| API Gateway |
| auth, rate limit |
+----+----+----+----+
| | |
----------------+ | +----------------
| | |
v v v
+----------------+ +----------------+ +----------------+
| Event Service | | Search Service | | Booking Service|
+--------+-------+ +--------+-------+ +---+--------+---+
| | | |
| | | |
v v v v
+----------------+ +----------------+ +------+ +----------------+
| Cache Redis | | Elasticsearch | |Redis | | Payment |
| event details | | full text | |Locks | | Processor |
+--------+-------+ +--------+-------+ | TTL | | Stripe |
| ^ +--+---+ +--------+-------+
| | | |
v | | |
+---------------------------------------------------------------+
| PostgreSQL |
| Events | Venues | Performers | Tickets | Bookings | Users |
+---------------------------------------------------------------+
^ ^
| |
+---------+----------+
|
CDC / sync
|
v
+------------------+
| Search indexer |
| or CDC pipeline |
+------------------+
Real-time updates for seat map
+----------------+
| Realtime/SSE |
| update service |
+-------+--------+
|
v
push seat status changes to clients
Optional protection for huge onsales
+----------------------+
| Virtual waiting queue|
| Redis sorted set |
+----------+-----------+
|
v
admits limited users to booking flow
```
The main idea is simple. Reads go through Event Service and Search Service, and writes with contention go through Booking Service. You cache event data for heavy reads, use Elasticsearch for fast search, use Redis TTL locks for temporary seat holds, and use PostgreSQL as the source of truth so you never double book.
If you want, I can also show you a smaller interview-ready version that is easier to draw in 2 minutes.
Stateful chat servers maintain WebSocket connections. A sends a message → server writes durably to Cassandra + creates Inbox entries → publishes to Kafka → B's server pushes via WebSocket. If B is offline the Inbox entry stays until B reconnects and acks.
Problem
Large-scale real-time chat. Let users send messages with very low delay, and still receive them later if offline.
Four concerns: group chats, fast delivery over persistent WebSocket, durable storage, and media sharing via blob storage.
Failures
**Chat server goes down with 100K active connections**
100K users instantly disconnected. All in-flight messages buffered on that server are potentially lost.
_Fix:_ Client auto-reconnects within 5s (exponential backoff). Messages written to Cassandra before delivery attempt — durability is in DB, not server memory. On reconnect, server replays pending Inbox entries.
**Kafka consumer falls behind (delivery lag)**
Messages appear delayed. Real-time feel is broken.
_Fix:_ Monitor consumer lag per partition. Add more delivery server consumers. Kafka retention is long enough (7 days) to replay. Alert at >1s lag.
**Group chat with 1,000 members — one message fans out to 1,000 servers**
One message may require 1,000 individual Kafka publishes + 1,000 delivery confirmations. Multiplied by 100 messages/min = massive overhead.
_Fix:_ Group chat has dedicated server affinity — all members of a group are hashed to a small set of servers. Reduces fan-out from 1,000 to ~10. At extreme scale, group server is a separate chat cluster.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 2B users, 100M DAU, avg 50 messages sent/day, avg group size 20 |
| Read QPS | Connection management: 100M persistent WebSocket connections — ~3,000 chat servers at 30K connections each |
| Write QPS | 100M × 50 / 86400 ≈ 58,000 message writes/s to Cassandra |
| Storage | 58K messages/s × 1KB avg × 86400 × 365 ≈ 1.8 PB/year — partitioned by conversation_id + time |
| Cache math | Active conversations in memory: 100M DAU × 20 active conversations × 100 bytes ≈ 200 GB server RAM across fleet |
| Verdict | Connection count (100M WebSockets) is the primary scaling challenge. 3,000 chat servers required just for connection management. |
Design decisions
**Redis Pub/Sub vs. Kafka for cross-server delivery**
→ Redis Pub/Sub for real-time routing + Cassandra Inbox for durability
Kafka has too much latency (>100ms) for real-time message delivery. Redis Pub/Sub is best-effort sub-10ms. Durability comes from Cassandra Inbox — separate concern from delivery speed.
_Revisit when:_ Could use a dedicated messaging fabric (like Erlang's BEAM, which WhatsApp actually uses) for better connection density.
**End-to-end encryption key management**
→ Signal Protocol — keys stored on device only, server is blind
Server never sees plaintext. No key escrow. This is also a product differentiator. Architectural implication: server can't do server-side search or content moderation of message text.
_Revisit when:_ Backup encryption keys require a separate key backup protocol (WhatsApp uses HSM-backed cloud key backup).
**Message ordering within a conversation**
→ Lamport timestamps from sending client + Cassandra clustering key
Server-side monotonic counters require coordination. Client timestamps + sequence numbers per conversation provide good-enough ordering without server bottleneck.
_Revisit when:_ For strict ordering, use per-conversation sequence numbers issued by the server (adds a write per message for the sequence counter).
Follow-up Q&A
**How do you handle a user who is offline for 30 days?**
Cassandra Inbox stores messages durably. On reconnect, server queries Inbox for all undelivered messages, pages them to client. After client acks, remove from Inbox. Retention policy: keep Inbox entries for 30 days, then expire.
**How do you route a message when you don't know which server the recipient is on?**
Service discovery: Redis hash(user_id) → server_id mapping. Updated on connect/disconnect. Message router looks up server_id, publishes to that server's Redis pub/sub channel. If user offline, write to Cassandra Inbox directly.
**How do you scale to 10B users?**
Horizontal: more chat servers. The key insight is that users who never talk to each other are independent — shard by user_id cluster. Regional deployment (EU users mostly talk to EU users). The design is embarrassingly parallel across conversation clusters.
**How do you handle media (photos, videos)?**
Media is never sent through chat servers. Client uploads directly to blob storage (pre-signed URL). Sends message with media URL. Recipients download directly from CDN. Chat server only carries a tiny metadata message.
**What's your read receipt / delivery receipt design?**
Three states: Sent (written to Cassandra), Delivered (client received and ACKed via WebSocket), Read (client opened conversation). Client sends ACK events back to server. Server fans ACK to sender's device. Stored per message in Cassandra.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: throughput, queue lag, cache effectiveness. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: async replication lag <30s; failover promotes read replica. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Single server. HTTP polling every 5s. SQLite messages. Works for a demo, breaks at 1,000 users.
**v2 — Real-time** — WebSocket chat servers. Cassandra for messages. Redis Pub/Sub for cross-server delivery. Inbox for offline delivery. Handles 10M users.
**v3 — 2B users** — 100M persistent connections across 3,000 servers. Group chat server affinity. E2E encryption. Regional deployment. Dedicated media pipeline.
Why it's hard to scale
The hard part is that WhatsApp mixes huge connection scale with strict delivery needs. You are not just storing chats. You are keeping hundreds of millions of long lived socket connections open, routing each message to the right server fast, and still making sure offline users eventually get every message.
There are three main scaling pain points. First, connection management is massive because every online user may hold one or more persistent connections, and those users are spread across many chat servers. Second, message routing gets tricky once sender and receiver are on different servers, so you need some way to bounce messages across the fleet without losing them. Third, delivery is both real time and durable. If Redis style pub sub drops a live message, the system still needs inbox storage, acks, reconnect logic, and sync to guarantee the user eventually gets it.
A fourth issue is fan out in group chats and multi device delivery. One message may need to reach many participants and several devices per participant, which multiplies writes and delivery work. So the simple interview answer is that WhatsApp is hard because it combines real time sockets, durable messaging, cross server routing, and offline sync all at once.
Key points
- **Mental model** — WebSocket for now, DB for later, blob storage for files.
- **Send path** — Write message durably first (Cassandra + Inbox), then try real-time delivery. Durability before speed.
- **Offline delivery** — Inbox entry persists until client acks. On reconnect, server replays pending messages.
- **Cross-server routing** — Redis Pub/Sub routes events to whichever server holds a user's connection. Best-effort — durability is in DB.
- **Attachments** — Pre-signed URL upload directly to blob storage. Server only stores the file reference.
> One sentence: Chat servers manage live WebSocket connections, Cassandra stores messages and inbox state, Redis Pub/Sub routes real-time events, blob storage handles media.
Tradeoffs
**WebSockets vs HTTP polling** — WebSockets are better for chat — low latency two-way. Tradeoff: harder to operate, many long-lived connections.
**Redis Pub/Sub vs Kafka for routing** — Redis is simpler and lighter for routing live events. Tradeoff: no delivery guarantee, so the DB-backed Inbox handles durability.
**At-least-once delivery vs exactly-once** — Exactly-once at messaging scale requires expensive coordination. At-least-once with client-side dedup (message_id) is the right tradeoff — simpler, faster, good enough.
**Per-device Inbox vs per-user Inbox** — Per-device inbox lets each device independently track delivery state and supports multi-device independently. Tradeoff: more rows, more complex fan-out on send.
> "Optimize for low-latency live delivery, but keep durability separate so failures in the real-time path don't lose messages."
Deep dives
#### Deep dive 1: Connection management and cross-server message routing at 100M concurrent WebSockets
> [!CAUTION]
> **🔴 Weak** — use a centralized message broker — every server publishes and subscribes to a shared queue
>
> [!WARNING]
> **🟡 Strong** — the core scaling pain is connection state: 100M online users each hold a persistent WebSocket to one of ~3,000 chat servers. A message from user A on server 1 to user B on server 3 must be routed without a central coordinator. Weak answer: Redis Pub/Sub. Strong answer: Redis Pub/Sub + per-user server affinity. Server affinity map: user_id → server_id stored in Redis hash (HSET). On connect: update map. On message send: look up recipient's server, publish to that server's channel. If recipient is offline: write to Cassandra Inbox directly
>
> [!TIP]
> **🟢 Staff+** — to name: what happens if the server holding a user's connection goes down? Client reconnects within 5 seconds (exponential backoff). Server affinity map is updated on reconnect. In-flight messages to the old server that weren't delivered: the Inbox ensures they're eventually delivered — real-time delivery is best-effort, durability is guaranteed by Cassandra. Never lose a message even when the delivery path fails
#### Deep dive 2: Message delivery guarantees — at-least-once with ack protocol
_WhatsApp's delivery promise: messages are never lost_
> [!CAUTION]
> **🔴 Weak** — write to DB, deliver, done
>
> [!WARNING]
> **🟡 Strong** — explicit ack protocol. When A sends a message: (1) server writes to Cassandra Inbox for B, (2) server attempts real-time delivery via WebSocket, (3) server returns delivery confirmation to A. B's client acks receipt: (4) client receives message, (5) sends ack to its chat server, (6) server marks Inbox entry as delivered, (7) server relays ack to A (double-checkmark). Read receipt: B opens the conversation, client sends read event, server relays to A (blue checkmark)
>
> [!TIP]
> **🟢 Staff+** — design: ack messages are small (just message_id + status) and can be batched. If B goes offline mid-conversation, Inbox stores pending messages. On reconnect: server queries Inbox WHERE user_id = B AND delivered = false, sends all pending, waits for acks before marking delivered. Message ordering: Cassandra clustering key on (conversation_id, created_at, message_id) gives total order within a conversation
#### Deep dive 3: Group chats — fan-out and multi-device delivery
> [!CAUTION]
> **🔴 Weak** — fan out one-by-one to all group members on every message. Naive approach: 1,000 individual deliveries per message. At 10 messages/min in an active group: 10,000 deliveries/min for one group
>
> [!WARNING]
> **🟡 Strong** — Weak answer: fan out one-by-one to all group members on every message. Naive approach: 1,000 individual deliveries per message. At 10 messages/min in an active group: 10,000 deliveries/min for one group
>
> [!TIP]
> **🟢 Staff+** — design: group chat server affinity — hash(group_id) to a dedicated set of chat servers. All group members' connections are preferentially routed to these servers. Fan-out from one message: server looks up group members, identifies which are connected to itself (direct push), which are on other servers in the affinity set (local pub/sub channel), which are offline (Cassandra Inbox). This reduces the fan-out from 1,000 individual cross-server calls to a broadcast within a small server cluster. Multi-device: Inbox is per-device, not per-user. Each device has its own Inbox entry and acks independently. Message is marked fully delivered only when all active devices have acked
_Why the deep dives connect to the scaling problem: "Huge connection scale, routing, durable messaging, and group fan-out." Each deep dive addresses one layer._
Interview script
1. Durability-first script.
2. "Clarifying questions: are we designing 1:1 messaging only, or also group chats? And what's the delivery guarantee — best-effort or durable?"
3. "Good — both 1:1 and groups, durable delivery. Core features: send message, deliver in real-time when recipient is online, queue for offline delivery, read receipts. Out of scope: voice/video, payments."
4. "Scale: 2B users, 100M DAU, ~50 messages/day per active user = 58K message writes/sec. The hard constraint is 100M persistent WebSocket connections."
5. "Architecture: stateful chat servers hold WebSocket connections. Each server handles ~30K concurrent connections — need ~3,000 chat servers. This is a connection management problem as much as a messaging problem."
6. "Message routing: when A sends to B, A's chat server looks up B's server in Redis (user_id → server_id). Publishes to that server's Redis Pub/Sub channel. B's server pushes to B's WebSocket. If B is offline: write directly to Cassandra Inbox."
7. "Delivery guarantee: Cassandra Inbox is the durability layer. On reconnect, server queries Inbox for undelivered messages and replays them. Client ACKs each message. Real-time delivery is best-effort; Inbox guarantees nothing is lost."
8. "Key tradeoff: Redis Pub/Sub is fast but has no delivery guarantee. That's fine because Cassandra Inbox is the fallback. Never rely on the real-time path alone for delivery correctness."
Whiteboard
```
+----------------------+
| Mobile and Web |
| Clients |
+----------+-----------+
|
WebSocket over TLS
|
+----------v-----------+
| L4 Load |
| Balancer |
+----------+-----------+
|
+------------------+------------------+
| |
+--------v--------+ +--------v--------+
| Chat Server | | Chat Server |
| A | | B |
|-----------------| |-----------------|
| conn map | | conn map |
| ack handling | | ack handling |
| heartbeat | | heartbeat |
| inbox sync | | inbox sync |
+---+---------+---+ +---+---------+---+
| | | |
| +-----------+ +-----------+ |
| | | |
| +------v---v------+ |
| | Redis Pub Sub | |
| | user channels | |
| +------+----------+ |
| | |
| | |
+--------v--------+ +--------v--------+ +--------v--------+
| Chat Table | | Message Table | | Inbox Table |
| chat metadata | | durable msgs | | undelivered per |
| by chatId | | by messageId | | user or client |
+-----------------+ +-----------------+ +-----------------+
\\\\ | /
\\\\ | /
\\\\ +---------v---------+ /
+----->| ChatParticipant |<------------+
| chatId, userId |
| + GSI by userId |
+-------------------+
Attachment flow
Client --get upload target--> Chat Server
Client --upload directly-----> Blob Storage
Client --send message with attachment URL--> Chat Server
Recipient --download with signed URL-------> Blob Storage
Delivery flow
```
FB News Feed Medium
Fan-out on write · Hybrid push/pull · Celebrity problem
Redis sorted setKafkaCassandraFan-out on writeHybrid push/pull
A new post writes to Cassandra then publishes to Kafka. An async fan-out worker prepends the post ID into each follower's Redis sorted set. Feed read = ZREVRANGE. Celebrity problem: accounts above a threshold are skipped at write time, pulled at read time and merged.
Problem
Showing each user a personalized list of recent posts from people they follow, fast and at massive scale.
The hard part: fan-out — one post can need to appear in millions of followers' feeds.
Failures
**Fan-out worker falls behind during a celebrity's viral post**
Millions of followers don't see the post in their feed for minutes. Feed appears stale.
_Fix:_ Celebrity threshold routing: any user above X followers bypasses fan-out entirely. Pull at read time. Worker backpressure handling with priority queues.
**A user's Redis feed sorted set grows unbounded**
ZREVRANGE on a set with 100K entries is slow. Memory grows for power users who follow thousands of people.
_Fix:_ Cap feed at last 1,000 post IDs per user. Trim on every fan-out write (ZREMRANGEBYRANK). Older posts are always fetchable from Cassandra.
**Hot post (viral content) DDOSes Cassandra**
One post_id is requested by millions of simultaneous users. Single Cassandra partition is hammered.
_Fix:_ Dedicated post cache layer (Redis or Memcached) in front of Cassandra. Viral post detection: if read QPS for a post_id exceeds threshold, promote to L1 cache.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 3B users, 10% DAU = 300M DAU, avg 5 feed reads/day, 1 post/week/user |
| Read QPS | 300M × 5 / 86400 ≈ 17,400 feed read QPS |
| Write QPS | 3B / 7 / 86400 ≈ 5,000 post write QPS → fan-out amplification: 5,000 × avg 200 followers = 1M feed write QPS |
| Storage | Feed cache: 3B users × 1,000 post IDs × 8 bytes ≈ 24 TB Redis — requires Redis cluster with sharding |
| Cache math | Celebrity with 100M followers × 1 post = 100M fan-out writes at 1M/s = 100 seconds to fan out. Too slow — pull model for celebs. |
| Verdict | Fan-out write amplification is the defining constraint. Hybrid model is not optional — it's mathematically required for celebrity accounts. |
Design decisions
**Fan-out on write vs. fan-out on read vs. hybrid**
→ Hybrid: push for normal users, pull for celebrities
Push-only: celebrity post → 100M writes → catastrophic. Pull-only: feed read → hundreds of DB queries to assemble timeline → too slow. Hybrid: threshold at ~100K followers (tunable) separates the two regimes.
_Revisit when:_ Threshold is a config value, not code. Adjust based on observed fan-out worker lag.
**Redis sorted set vs. Cassandra-backed feed**
→ Redis sorted set per user (score=timestamp)
Feed reads must be sub-100ms. Cassandra read latency ~1ms but requires complex fan-in query. Redis ZREVRANGE is O(log N + K) from in-memory — consistently sub-ms.
_Revisit when:_ For low-activity users (follow few people), on-demand Cassandra query is cheaper than maintaining a Redis entry.
**Ranking: chronological vs. ML**
→ Chronological for v1, ML ranking layer on top
Chronological is simple and understandable. ML ranking is a separate concern — it sits above the retrieval layer and re-scores candidates before serving. Don't conflate retrieval with ranking in the design.
_Revisit when:_ Always run ML ranking as a separate service to avoid coupling.
Follow-up Q&A
**How do you handle a user who follows 10,000 people?**
Fan-out on write to all 10K is expensive but one-time per post. The real issue is feed read: ZREVRANGE across 10K followed users' posts merged together. Solution: pre-merge into the user's own sorted set at write time. Reading is always O(1) regardless of follow count.
**What happens if the fan-out worker crashes mid-fan-out?**
Kafka provides at-least-once delivery. Fan-out workers are idempotent — writing the same post_id twice to a Redis sorted set with the same score is a no-op. On restart, worker replays from last committed Kafka offset.
**How do you serve the feed if Redis is unavailable?**
Fallback to on-demand Cassandra assembly with a lower quality (higher latency) feed. Acceptable degradation. Redis failure should be rare with proper replication.
**How do you handle edits or deletions of posts?**
Soft delete: mark post as deleted in Cassandra. Feed service filters deleted posts at read time. Don't try to fan-out deletes — it's the same amplification problem. Users may briefly see deleted posts until their feed refreshes.
**How would you add ranked (non-chronological) feed?**
Keep retrieval layer unchanged — Redis still stores post_id candidates. Add ranking service: takes top-N candidates from Redis, fetches features (engagement, recency, relationship strength), applies ML model, returns re-scored top-K. Separate concern, separate service.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: throughput, queue lag, cache effectiveness. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: async replication lag <30s; failover promotes read replica. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Fan-out on read: SELECT posts FROM follows WHERE user_id IN (...) ORDER BY created_at. Works up to ~100 follows per user. Breaks at scale.
**v2 — Fan-out on write** — Kafka + async fan-out workers. Redis sorted sets per user. Handles normal users. Celebrity accounts still problematic.
**v3 — Hybrid + ranking** — Hybrid fan-out with celebrity threshold. ML ranking layer. Feed capped at 1,000 posts. Viral post detection and L1 caching. Hot key protection.
Why it's hard to scale
The hard part is fan-out. In FB News Feed, one user action can explode into huge work either when you read the feed or when you write a new post.
There are two core scaling pain points. First, feed reads can be expensive because to build one timeline you may need posts from a very large number of followed users, then merge and sort them fast. That is fan-out on read. Second, feed writes can also be expensive because a user with millions of followers may force you to update millions of feeds when they post. That is fan-out on write.
A third issue is skew. Most posts are quiet, but a few become very hot, so one post or one celebrity account can create uneven load on your databases and caches. So the big idea you should say in an interview is that News Feed is hard because of massive fan-out, hot keys, and the trade-off between precomputing feeds for fast reads versus computing them later for cheaper writes.
Key points
- **Goal** — Show each user a personalized list of recent posts from people they follow.
- **Default scaling move** — Precompute feeds on write. Store a bounded recent feed per user in Redis sorted set.
- **Write path** — New post → Cassandra → Kafka → workers update follower feeds asynchronously.
- **Celebrity fix** — Don't fan out to huge accounts' followers. Pull their recent posts at read time and merge.
- **Hot post fix** — Cache in front of post storage so viral posts don't hammer the DB.
> Mental model: hybrid fan-out. Normal users = fan-out on write. Celebrity users = fan-out on read.
Tradeoffs
**Fan-out on write vs fan-out on read** — Fan-out on write: fast feed reads, expensive for users with many followers. Fan-out on read: cheap writes, slow reads at scale.
**Chronological vs ML-ranked feed** — Chronological is simple, predictable, no model needed. ML ranking improves engagement but adds a two-stage retrieval+scoring pipeline and requires feature infrastructure.
**Cassandra vs PostgreSQL for post store** — Cassandra handles write-heavy append-only workloads well — posts are written once, read many times, partitioned by user. PostgreSQL struggles at Cassandra-scale fan-out writes.
**Bounded feed (top 1K) vs unbounded** — Unbounded Redis sorted set per user grows forever — memory cost becomes prohibitive. Capping at 1K entries with ZREMRANGEBYRANK bounds cost; older posts are always fetchable from Cassandra.
> "Use hybrid fan-out. Precompute for normal users, pull at read time for celebrities."
Deep dives
#### Deep dive 1: Fan-out architecture — the push/pull/hybrid decision
> [!CAUTION]
> **🔴 Weak** — always fan-out on write — pre-compute every follower's feed on every post
>
> [!WARNING]
> **🟡 Strong** — the core hard problem is fan-out: one post potentially needs to update millions of feeds. Weak answer: always fan-out on write. Strong answer: articulate the full tradeoff space. Fan-out on write: fast reads (pre-computed), expensive writes (proportional to follower count). Fan-out on read: cheap writes, expensive reads (O(followed_users) per feed request). Hybrid: fan-out on write for normal users, pull on read for celebrities above a threshold
>
> [!TIP]
> **🟢 Staff+** — the threshold is a configuration value, not a code decision — it should be tunable based on observed fan-out worker lag. The math: if a fan-out worker processes 10K writes/second and delivery SLA is 5 minutes, the maximum safe follower count for push fan-out is 10K × 300 = 3M. Any account above that threshold uses pull. At read time: fetch user's pre-built feed from Redis, also fetch the last N posts from each celebrity they follow from Cassandra, merge and sort. The merge is O(C × log C) where C is the number of celebrity accounts — typically small, so fast
#### Deep dive 2: Feed storage and retrieval — Redis sorted sets at petabyte scale
> [!CAUTION]
> **🔴 Weak** — query Cassandra directly on every feed read — join followed users' posts and sort
>
> [!WARNING]
> **🟡 Strong** — each user has a Redis sorted set keyed by user_id with post_ids as members and publish_timestamp as score. ZREVRANGE returns the feed in reverse-chronological order in O(log N + K) time
>
> [!TIP]
> **🟢 Staff+** — concerns: (1) memory: 3B users × 1,000 post IDs × 8 bytes = 24 TB of Redis storage — requires a large Redis cluster with sharding by user_id. Cost is significant — only store the most recent 1,000 posts per user (ZREMRANGEBYRANK after every ZADD to trim). (2) Hot keys: users followed by millions of other users have their post IDs written to millions of sorted sets concurrently — this is the fan-out bottleneck, not the sorted set read. (3) Cold users: users who haven't logged in for 30 days don't need a live Redis feed — evict cold feed entries, rebuild from Cassandra on next login
#### Deep dive 3: ML ranking — candidate retrieval vs. scoring separation
_In production, chronological feed is deprecated in favor of ML-ranked feed_
> [!CAUTION]
> **🔴 Weak** — Query the database on every feed request.
>
> [!WARNING]
> **🟡 Strong** — In production, chronological feed is deprecated in favor of ML-ranked feed
>
> [!TIP]
> **🟢 Staff+** — this is a two-stage architecture. Stage 1 (retrieval): pull top-N candidates from Redis sorted set — fast, based on recency signal only. Stage 2 (ranking): pass candidates to a separate ranking service that scores each post using features (engagement rate, relationship strength, content type, recency, user interest signals). Serves top-K ranked results. These two stages are explicitly separate services with separate scaling characteristics: retrieval is read-heavy and latency-critical, ranking is compute-heavy and can tolerate 50-100ms. Staff+ architectural point: never embed ML ranking logic in the feed service — they change at different cadences, are owned by different teams, and have different failure modes. The interface is clean: retrieval service returns candidates, ranking service returns scores
_Why the deep dives connect to the scaling problem: "Massive fan-out and hot keys." Deep dive 1 solves fan-out architecture. Deep dive 2 solves storage and retrieval. Deep dive 3 solves the product quality layer on top._
Interview script
1. Fan-out-first script.
2. "Clarifying questions: are we designing for a social network at Facebook scale — billions of users, celebrity accounts with hundreds of millions of followers? And is the feed chronological or ranked?"
3. "Good — FB scale, ranked feed. Core features: user creates post, feed shows recent posts from followed accounts ranked by relevance. Out of scope: Stories, Groups, ads injection (unless asked)."
4. "Scale: 3B users, 300M DAU, 5K post writes/sec, 17,400 feed read QPS. The defining constraint: fan-out. One post to an account with 50M followers = 50M Redis writes. That's the design problem."
5. "Write path: post → Cassandra (durable) → Kafka → fan-out workers → push post_id to each follower's Redis sorted set. Fast reads: ZREVRANGE is O(1) per feed read."
6. "Celebrity problem: any account above ~1M followers bypasses fan-out. At read time, pull their last N posts from Cassandra and merge with the pre-built feed. Merge cost is O(C × log C) where C is the number of followed celebrity accounts — typically < 10."
7. "ML ranking: two-stage. Stage 1 (retrieval): ZREVRANGE from Redis — fast, recency-sorted, returns top-1000 candidates. Stage 2 (ranking): separate service scores each candidate using engagement signals, relationship strength, content type. Returns top-50. Keep these stages strictly separate — they change at different cadences."
8. "Key failure mode to name: fan-out worker falls behind during a viral event. Fix: celebrity threshold is a config value — lower it dynamically when worker lag exceeds 2 minutes. This is the operational lever that keeps the system stable under load."
Whiteboard
```
Clients
|
v
API Gateway / Load Balancer
|
+-------------------+-------------------+-------------------+
| | |
v v v
Post Service Follow Service Feed Service
| | |
| | |
v v v
Post Table Follow Table Precomputed Feed Table
DynamoDB DynamoDB DynamoDB
PK postId PK userFollowing PK userId
SK userFollowed value recent postIds
GSI userFollowed
Post Table GSI
PK creatorId
SK createdAt
Write path for new post
-----------------------
User -> Post Service -> Post Table
-> Queue message with postId, creatorId
v
SQS / Queue
|
v
Feed Workers
|
+--------------+--------------+
| |
v v
Follow Table GSI Precomputed Feed Table
get followers prepend new postId
Read path for feed
------------------
User -> Feed Service
-> read precomputed feed for user
-> for non-precomputed celebrity accounts, query recent posts by creatorId from Post Table GSI
-> fetch post objects by postId
-> merge + sort by createdAt
-> return page with next cursor
Hot post read protection
------------------------
Feed Service
|
v
Replicated Redis Cache
|
v
Post Table
```
If you want the best interview version, I would say this out loud as one sentence. Most users read from a precomputed feed, most posts are fanned out asynchronously on write, and celebrity accounts fall back to partial fan-out on read.
If you want, I can also give you a smaller interview-sized sketch that fits in 10 to 12 lines.
Tinder — dating app Medium
Geospatial deck · Like detection · Match notification
Swipe deck is pre-computed via GEORADIUS + preference filters, cached per user. On a like: write to Cassandra + atomically check if the other person already liked back. Mutual = match → WebSocket push to both users.
Problem
Tinder solves real-time recommendation and mutual match detection. Show relevant nearby profiles fast, record huge swipe volumes, and reliably detect when two users both say yes.
Failures
**Redis GEO index too large for a dense city**
GEORADIUS in a city with 10M users returns too many candidates. Filter step is expensive.
_Fix:_ Narrow GEORADIUS radius. Apply age/preference filters at Redis level using sorted set intersection. Pre-filter by coarse geohash first.
**Swipe history grows unbounded per user**
Checking 'have I already swiped this profile?' requires scanning a large set. Slow.
_Fix:_ Bloom filter per user for seen profiles. O(1) check, small memory. False positives = occasionally hiding an unseen profile (acceptable). Exact check for match creation only.
**Match notification fails to deliver**
User swipes right, their match already swiped right, but neither gets notified. Match silently lost.
_Fix:_ Write match to Cassandra first. Push notification is best-effort but match is durable. On app open, always sync pending matches from Cassandra. Never rely on real-time delivery alone for match creation.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 50M DAU, avg 100 swipes/day, 1 match per 100 swipes |
| Read QPS | Deck generation: 50M × 10 deck requests/day / 86400 ≈ 5,800 deck QPS |
| Write QPS | 50M × 100 swipes / 86400 ≈ 58,000 swipe write QPS — Cassandra's sweet spot |
| Storage | 58K swipes/s × 86400s × 365 days × 100 bytes ≈ 183 TB/year of swipe history |
| Cache math | Pre-computed deck: 50M users × 100 profiles × 200 bytes ≈ 1 TB Redis — requires Redis cluster, but feasible. |
| Verdict | Swipe writes are the dominant workload. Cassandra is the right choice. Deck pre-computation is memory-heavy but essential for sub-second UX. |
Design decisions
**Pre-compute deck vs. real-time generation**
→ Pre-compute deck, refresh async
Real-time GEORADIUS + preference filter + swipe history dedup on every swipe takes too long. Pre-compute a stack of 50-100 candidates, serve instantly, refresh in background when stack is low.
_Revisit when:_ Real-time generation acceptable if filter set is small (sparse area, few active users nearby).
**Redis atomic check vs. Cassandra for match detection**
→ Redis SETNX for atomic mutual check + Cassandra for durability
Two users can swipe simultaneously. Need atomic operation to detect mutual like. Redis SETNX: set key 'match:min(a,b):max(a,b)' — if already set, it's a match. Cassandra stores durably.
_Revisit when:_ If Redis is unavailable, fall back to Cassandra with conditional writes (LWT).
**Bloom filter vs. exact set for swipe history dedup**
→ Bloom filter for deck generation, exact Cassandra for match detection
Bloom filter for deck: O(1), ~10 bytes/entry, 1% false positive rate acceptable (occasionally hide an unseen profile). For match: must be exact — false negative (missing a match) is unacceptable.
_Revisit when:_ Cuckoo filter instead of Bloom for deletable entries (users can revoke swipes in some markets).
Follow-up Q&A
**How do you handle a user who moves to a new city?**
Update Redis GEOADD with new location. Invalidate pre-computed deck (now stale — wrong city). Async re-generate deck for new location. Deck generation is triggered by location change events, not just swipes.
**What happens if two users swipe on each other simultaneously?**
Redis SETNX is atomic. First SETNX sets the key. Second SETNX finds key present = match detected. No race condition. Both users get notified via their respective response objects.
**How do you prevent running out of profiles in a small market?**
Track deck depth. When < 10 profiles remain, async re-generate. Widen radius or relax preference constraints progressively. Show 'no more profiles in your area' only as last resort.
**How does the recommendation system affect this design?**
ML model is a separate concern — it scores candidate profiles by predicted swipe probability. Add a scoring step after GEORADIUS but before serving deck. Score computed async, stored per (user_id, candidate_id). Deck is pre-scored candidates sorted by ML score, not just distance.
**How would you add a rewind feature (undo last swipe)?**
Soft-delete swipes in Cassandra (status = PENDING_UNDO). Remove from Bloom filter (use Cuckoo filter instead). Re-add profile to deck head. Match is undone only if the other user hasn't matched back yet.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: throughput, queue lag, cache effectiveness. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: async replication lag <30s; failover promotes read replica. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — SQL with PostGIS. Swipe history in Postgres. Match detection with DB unique constraint. Real-time deck generation. Works up to ~100K users per city.
**v2 — Scale** — Cassandra for swipe writes. Redis GEO for location. Pre-computed deck with Bloom filter dedup. Redis SETNX for atomic match detection. Push notifications via APNS/FCM.
**v3 — Personalize** — ML scoring layer for deck ranking. Smart radius expansion for sparse markets. ELO-style attractiveness score. Super Likes as separate flow with stricter rate limiting.
Why it's hard to scale
The hard part is that Tinder has both a fast read problem and a correctness problem. You need to generate a fresh stack of nearby profiles in well under a second, while also making sure swipes and matches are recorded correctly.
There are three main scaling pain points. First, feed generation is expensive because it mixes filters like age and preferences with geospatial search, which means a simple database query gets slow fast at large scale. Second, swiping is a huge write stream, and match creation is tricky because two users can swipe on each other at nearly the same time, so you need low latency plus strong enough consistency to not miss a match. Third, you must avoid re-showing profiles a user already swiped on, which gets harder as each user builds a large swipe history.
A good mental model is this. Tinder is hard because it combines search, real-time decisions, and deduping in one loop. You are not just storing profiles. You are constantly finding nearby candidates, filtering out old ones, and atomically detecting mutual likes.
Key points
- **Scope it first** — Core: swipe, match, chat stub, nearby profiles. Out of scope unless asked: payments, Super Likes, video, reporting. Geo is central — say so upfront.
- **Build the deck fast** — Pre-computed candidate stack per user. Async re-fill from GEORADIUS + preference filter when stack runs low. Never compute deck on every swipe.
- **High write volume — Cassandra** — 58K swipe writes/sec. Cassandra append-only, partitioned by user_id — the right DB for this access pattern. Never use PostgreSQL for raw swipe writes at this scale.
- **Instant match detection — Redis SETNX** — SETNX on key min(a,b):max(a,b). Atomic. First caller creates the key. Second caller finds it exists = match. No race possible.
- **Swipe dedup — Bloom filter** — 73K swipes/user over 2 years. Bloom filter: O(1) check, ~730 KB per user. 1% false positive acceptable for deck (skip an unseen profile). Exact Cassandra check only for match creation.
- **Privacy by design** — Never expose who liked you without a mutual match. The SETNX key reveals nothing — it only exists on mutual. The pending swipe in the other direction is never returned by any API.
- **Failure mode to name** — Redis goes down → fall back to Cassandra LWT (Lightweight Transaction) for match detection. Slower but correct. Always have a fallback for the atomic match check.
> Frame as three problems: low-latency feed (pre-computed deck), high-volume swipe writes (Cassandra), instant match detection (Redis atomic). Name all three in your opening.
Tradeoffs
**Cached deck vs real-time search** — Cached deck is fast but stale. Real-time search is fresher but slower. Answer: cached candidates with search index refill.
**Redis for match vs Cassandra only** — Cassandra alone can miss near-simultaneous mutual likes. Redis gives atomic match detection.
**Bloom filter vs exact set for swipe dedup** — Bloom filter: O(1) check, ~10 bytes/entry, 1% false positive (occasionally hides an unseen profile — acceptable). Exact Cassandra set: correct but O(log N) per check at 58K swipes/sec. Use Bloom for deck, exact for match.
**Push notifications vs in-app WebSocket for match** — Push (APNs/FCM) reaches users when app is backgrounded. WebSocket is lower latency when app is open. Both are needed — WebSocket for active session, push as fallback.
> Deck: cache wins over freshness. Match: Redis atomic over Cassandra-only. Swipe storage: Cassandra over PostgreSQL. Each decision prioritizes the dominant access pattern for that flow.
Deep dives
#### Deep dive 1: Recommendation stack — low-latency candidate generation
> [!CAUTION]
> **🔴 Weak** — run GEORADIUS + preference filter on every swipe in real-time
>
> [!WARNING]
> **🟡 Strong** — the scaling pain is that feed generation mixes geospatial search, preference filtering, and swipe history deduplication into one pipeline that must complete in under a second. Weak answer: GEORADIUS + filter in one query. Strong answer: pre-compute candidate decks asynchronously — GEORADIUS query + preference filter runs in background, results cached per user. When user opens the app, deck is ready instantly
>
> [!TIP]
> **🟢 Staff+** — the deck has a TTL (location changes, preferences update). Rather than a fixed TTL, invalidate the deck reactively: location update event → deck invalidation → async re-generation. Stack depth monitoring: when user swipes through to the last 10 profiles in their deck, trigger an async deck refresh before they run out. Candidate scoring: after GEORADIUS + preference filter, score remaining candidates using ML model (predicted swipe probability based on historical data) and serve the highest-scoring profiles first
#### Deep dive 2: Swipe storage and match detection — correctness under concurrent writes
_58,000 swipe writes/second to Cassandra with one correctness requirement: two simultaneous mutual likes must be detected exactly once_
> [!CAUTION]
> **🔴 Weak** — check Cassandra for mutual like
>
> [!WARNING]
> **🟡 Strong** — Redis SETNX on a derived key (min(a,b):max(a,b)) for atomic mutual check. If A likes B and B already liked A, the key exists → match. If A likes B and B hasn't liked yet, SETNX succeeds → key set for later. When B likes A, SETNX finds the key → match detected
>
> [!TIP]
> **🟢 Staff+** — durability: Redis is not the source of truth. Cassandra stores all swipes durably. Redis is only for the real-time match detection signal. If Redis is down: fall back to Cassandra CAS (Compare-And-Swap using Lightweight Transactions) — slower but correct. Partition key for swipes: (user_id, created_at_bucket) — queries like "all swipes by user X" are efficient
#### Deep dive 3: Avoiding re-shows — Bloom filter for swipe history
> [!CAUTION]
> **🔴 Weak** — store all swipe history in Cassandra, check each candidate with a point query. Each user has accumulated swipe history that must be filtered from their deck. At 100 swipes/day × 2 years = 73,000 swipes per active user. Checking all 73K against candidate profiles is expensive
>
> [!WARNING]
> **🟡 Strong** — Weak answer: store all swipe history in Cassandra, check each candidate with a point query. Each user has accumulated swipe history that must be filtered from their deck. At 100 swipes/day × 2 years = 73,000 swipes per active user. Checking all 73K against candidate profiles is expensive
>
> [!TIP]
> **🟢 Staff+** — solution: Bloom filter per user (probabilistic, O(1) check, ~10 bytes/entry at 1% false positive rate). 73K swipes × 10 bytes = 730 KB per user — stored in Redis. False positive means occasionally hiding an unseen profile (user never sees it) — this is acceptable, better than showing an already-swiped profile. Critical distinction: Bloom filter is for deck generation only. For match detection, exact Cassandra check is required — a false negative (missing a match) is unacceptable. Cuckoo filter as an upgrade: supports deletions (allowing swipe undo) at slightly higher memory cost
_Why the deep dives connect to the scaling problem: "Fast read, correctness, and deduplication in one loop." Deep dives address each dimension of that loop._
Interview script
1. Three-problem script.
2. "Clarifying questions: are we designing for the core swipe + match flow, or also chat? And is the deck generation per-session or pre-computed?"
3. "Good — swipe, match, and chat stub. I'd frame this as three distinct problems: deck generation (low latency), swipe storage (high volume), match detection (correctness under concurrency)."
4. "Scale: 50M DAU, 100 swipes/day = 58K swipe writes/sec. Deck generation: 5,800 requests/day per user but needs sub-100ms response."
5. "Deck generation: pre-compute per user async. GEORADIUS finds candidates within radius, filter by age and preferences, score by ML model, cache in Redis. On app open, deck is ready. Async re-fill when stack drops below 10 profiles."
6. "Swipe storage: Cassandra, partitioned by user_id and time bucket. Write-optimized. 58K writes/sec is well within Cassandra's sweet spot."
7. "Match detection: Redis SETNX on key min(a,b):max(a,b). Both users swipe right — second SETNX finds the key already set, that's the match. Atomic, no race. Cassandra stores all swipes durably — Redis is only for the real-time detection signal."
8. "Bloom filter for swipe dedup: each user accumulates ~73K swipes over 2 years. Bloom filter: O(1) check, ~730 KB per user in Redis. Prevents re-showing profiles without scanning the full history."
Whiteboard
```
+-------------------+
| Mobile Client |
| profile, feed, |
| swipe, match UI |
+---------+---------+
|
HTTPS |
v
+-------------------+
| API Gateway |
| auth, routing |
+----+---------+----+
| |
+------------+ +-----------------+
| |
v v
+---------------------+ +----------------------+
| Profile Service | | Swipe Service |
| prefs, profile data | | record swipe, detect |
+----------+----------+ | match, emit events |
| +----+-----------+-----+
| | |
v | |
+---------------------+ | v
| User DB | | +------------------+
| profiles, prefs | | | Notification Svc |
+---------------------+ | | APNS or FCM |
| +--------+---------+
| |
| v
| +---------------+
| | Other User |
| | push device |
| +---------------+
|
v
+----------------------+
| Redis Match Store |
| atomic pair check |
| low latency match |
+----------+-----------+
|
v
+----------------------+
| Swipe DB |
| Cassandra style |
| durable swipe log |
+----------------------+
Feed generation path
+---------------------+
| Feed Service |
| build candidate set |
+-----+---------+-----+
| |
| v
| +----------------------+
| | Feed Cache |
| | precomputed stacks |
| +----------------------+
|
v
+-------------------------+
| Search Index |
| geo plus preference |
| filtering |
+-----------+-------------+
|
v
+---------------------+
| User DB / CDC sync |
| profile updates flow |
| into search index |
+---------------------+
```
The mental model is two main paths. One path serves profiles fast through feed cache plus a search index. The other path handles swipes safely through Redis for atomic match detection and Cassandra for durable swipe history.
If you want, I can also give you a simpler interview version with only 6 boxes, or a step by step swipe flow sketch.
Submit → enqueue in SQS → worker pulls job → runs code inside a Docker container (no network, CPU/memory limits, killed on timeout) → writes verdict to Redis. Client polls for result. Leaderboard = Redis sorted set (ZINCRBY/ZREVRANGE).
Problem
Safe, fast code evaluation at scale. Users browse problems, submit code, and get feedback quickly while the platform safely runs untrusted code.
The real challenge: executing user code in isolation, not storing problems.
Failures
**Execution worker pool exhausted during contest start**
All submissions queue up. Users wait minutes for results. Contest rankings are delayed.
_Fix:_ Pre-scale worker pool 30 min before contest (predictable traffic pattern). Autoscale by queue depth with aggressive scale-up policy. Shed load gracefully: queue max depth, return 'system busy, retry in 30s' rather than timing out silently.
**Container escapes resource limits (memory bomb)**
Single submission kills a worker node, slowing all other submissions on that node.
_Fix:_ cgroups + seccomp profile. Hard OOM kill at container level. Node-level memory pressure monitoring — evict container if host reaches 90% memory. Isolate contest traffic on dedicated node pool.
**Leaderboard Redis sorted set becomes hot during live contest**
Thousands of users polling leaderboard every 10s = massive ZREVRANGE QPS on one key.
_Fix:_ Cache leaderboard snapshot (CDN or in-process cache) with 5s TTL. Push updates via SSE instead of polling. Rate-limit leaderboard endpoint per user.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 500K DAU normal, 100K concurrent during contest, avg 5 submissions/user/contest |
| Read QPS | 100K users × 1 poll/10s = 10,000 leaderboard QPS during contest peak |
| Write QPS | 100K × 5 submissions / (2hr × 3600) ≈ 69 submission QPS normal, spikes to 5,000 at contest open |
| Storage | Each submission: code (50KB) + test output (10KB) × 100M submissions = 6 TB — S3 + metadata in PG |
| Cache math | 5,000 submission QPS × 10s execution time = 50,000 concurrent containers needed at peak. Reserved capacity + spot instances. Pre-warmed container pool. |
| Verdict | The submission burst at contest open is the design constraint. Not steady-state QPS but the 60-second spike at T=0. |
Design decisions
**Docker containers vs. VMs vs. AWS Lambda**
→ Docker with seccomp on dedicated worker nodes
Lambda cold start ~100ms is too slow. VMs ~5s startup is too slow. Pre-warmed Docker containers start in <100ms with no cold start. Dedicated nodes prevent noisy neighbor problems.
_Revisit when:_ Firecracker microVMs (AWS's approach) give VM-level isolation at near-container speed. Best of both worlds for security-critical execution.
**Redis for result storage vs. PG**
→ Redis with TTL for ephemeral results, PG for permanent record
Client polls result ID for ~10 seconds. Redis handles this trivially. No need to query PG on every poll. After 1hr, result moves to PG for permanent storage.
_Revisit when:_ WebSocket push instead of polling would eliminate Redis as a result buffer entirely.
**Single queue vs. per-language queues**
→ Per-language worker pools behind a shared queue
Python is 10× slower than C++. Mixed queue means Python submissions starve C++ workers. Separate queues: Python, JavaScript, C++/Java/Go each get dedicated worker pools sized by submission volume.
_Revisit when:_ Start with single queue for simplicity. Split only when language-specific tail latency becomes a user complaint.
Follow-up Q&A
**How do you prevent user code from making network calls?**
Docker network namespace with --network none. seccomp profile blocks socket syscalls. No iptables needed — network namespace isolation is at kernel level. Test this explicitly — it's the most critical security property.
**How do you handle an infinite loop submission?**
Hard CPU timeout via cgroups (e.g., 10 seconds CPU time, not wall time). Container killed and result set to TLE (Time Limit Exceeded). PID limits prevent fork bombs. Memory limit prevents memory exhaustion.
**How do you test new language versions or judge upgrades?**
Canary deployment: route 5% of submissions to new judge version. Compare results against current version. Roll back if disagreement rate > 0.1%. Never upgrade judges during live contests.
**How do you support user-defined test cases?**
Store user test cases in S3. Worker downloads test case bundle before execution. Isolate user test cases from canonical test cases — different output comparison logic. Rate-limit custom test case execution (more expensive than standard submission).
**What's your SLA for submission results?**
P50 < 3s, P99 < 30s. Anything beyond 30s should re-queue, not timeout. Users should never see a blank result — always a status (Queued / Running / Accepted / TLE / etc).
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: throughput, queue lag, cache effectiveness. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: async replication lag <30s; failover promotes read replica. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Single server running code in subprocess with timeout. No isolation. PG for everything. Works for low traffic, unacceptable security risk.
**v2 — Isolated execution** — Docker containers with resource limits. SQS job queue. Redis for results. Worker autoscaling. Handles normal traffic safely.
**v3 — Contest scale** — Pre-warmed container pools. Per-language queues. Leaderboard with SSE push. Dedicated contest node pool. Contest-time traffic isolation from normal users.
Why it's hard to scale
The hard part is not storing problems. It is safely running huge bursts of user code while still returning results fast.
There are three main scaling pain points. First, code execution is CPU heavy and spiky, especially during contests, so a wave of submissions can overwhelm workers much faster than normal page reads would. Second, every submission is untrusted code, so you need strong isolation, timeouts, and resource limits, which makes execution slower and more expensive than a normal backend request. Third, live leaderboards can create a read storm if thousands of users poll every few seconds, so you do not want to rebuild rankings from the main database on every request.
A good mental model is this. LeetCode is hard because it mixes a fairly simple content app with a mini compute platform. The content side is easy to scale. The code runner and contest traffic are the parts that make it tricky.
Key points
- **Scope it first** — Core: submit code, execute safely, return verdict (Accepted/TLE/WA/RE), leaderboard for contests. Out of scope: code completion, collaboration, plagiarism detection.
- **Submissions are async — always** — Never execute user code in the request thread. POST /submit returns a submission_id immediately. Client polls GET /submissions/:id. Queue + workers is the only correct pattern.
- **Sandbox is the hard part** — Docker + seccomp profile: block socket syscalls (no network), hard CPU + memory cgroups, OOM kill at container level, read-only filesystem, PID limit. Name all four constraints.
- **Container pre-warming** — Cold-start Docker container: ~100ms. Pre-warm a pool sized to P99 submission rate. For contests: pre-scale 30 min before start. Autoscale by queue depth, not CPU.
- **Per-language worker pools** — Python submissions take 5-10× longer than C++. Single mixed queue starves fast languages. Separate pools per language, sized by submission volume mix.
- **Leaderboard — Redis sorted set** — ZADD contest:{id} score user_id. ZREVRANK for user rank in O(log N). Redis handles 10K leaderboard QPS trivially. SSE push for live updates — never poll.
- **Failure mode to name** — Worker crashes mid-execution: job requeues via SQS visibility timeout. Duplicate execution is safe because verdict is deterministic — same code + same tests always gives same result.
> Lead with the sandbox — it's what makes this system unusual. 'The hard part isn't the queue, it's safely running untrusted code.' Name seccomp, cgroups, and no-network in your opening.
Tradeoffs
**Containers vs VMs** — Containers are faster and cheaper. Tradeoff: need careful seccomp sandboxing. VMs are safer but slower.
**Queue + async vs sync execution** — Queue is more reliable under spikes and gives retries. Tradeoff: client must poll for result.
**Pre-warmed pool vs cold-start containers** — Pre-warmed containers eliminate cold-start latency (~100ms saved) but waste compute when idle. For contest traffic with predictable spikes, pre-warming is worth it. For steady-state, autoscale from zero.
**Per-language queues vs single queue** — Single queue is simpler. Per-language queues prevent slow Python submissions from starving fast C++ workers — tail latency is dramatically better with language affinity.
> Containers over VMs for speed and cost. Pre-warmed pools over cold-start for contest UX. Per-language queues over single queue for fair tail latency. Async over sync because execution takes seconds.
Deep dives
#### Deep dive 1: Secure code execution — isolation, resource limits, and the sandbox design
_The defining problem is safely running untrusted code_
> [!CAUTION]
> **🔴 Weak** — Docker containers
>
> [!WARNING]
> **🟡 Strong** — Docker with specific security configuration: (1) seccomp profile (restrict allowed syscalls to the minimum needed — block socket, fork beyond a count, exec of new binaries), (2) no network namespace (--network none), (3) read-only filesystem, (4) cgroups for CPU time limit and memory limit, (5) OOM kill at container level
>
> [!TIP]
> **🟢 Staff+** — VMs give stronger isolation (separate kernel) at the cost of 5-10s startup vs. Docker's <100ms. Firecracker microVMs (used by AWS Lambda) give VM-level isolation at near-container startup speed — the best security/performance tradeoff. For a real coding judge: Docker + seccomp is the standard production answer. Name what your seccomp profile blocks explicitly: socket syscalls (no network), fork/clone beyond the PID limit (no fork bombs), mount syscalls (no filesystem escapes)
#### Deep dive 2: Handling submission spikes — contest traffic at 5,000 QPS
_Contest submissions: 100K users × 5 submissions in 2 hours = 250K total. In the first 10 minutes of a contest, most submissions happen → spike to 5,000 QPS_
> [!CAUTION]
> **🔴 Weak** — scale workers
>
> [!WARNING]
> **🟡 Strong** — SQS queue decouples submission acceptance from execution. API accepts submission immediately (synchronous, <10ms), enqueues job, returns submission ID. Client polls GET /submissions/:id. Queue depth × avg execution time / worker count = queue latency. At 5,000 QPS with 10s avg execution and 200 workers: 5,000 × 10 / 200 = 250 second latency at peak. Fix: pre-scale workers before contest start (predictable traffic pattern). Worker autoscaling metric: SQS queue depth, not CPU
>
> [!TIP]
> **🟢 Staff+** — separate worker pools by language (Python workers, C++ workers) because Python submissions take 5-10× longer than C++ — mixed queue starves C++ users. Per-language queue with proportional worker allocation
#### Deep dive 3: Leaderboard at real-time scale — Redis sorted sets under contest stress
_During a contest, 100K users polling leaderboard every 10 seconds = 10K QPS on one sorted set (the leaderboard is a single ZREVRANGE key)_
> [!CAUTION]
> **🔴 Weak** — cache the leaderboard
>
> [!WARNING]
> **🟡 Strong** — tiered caching — Redis sorted set is the live source, but serve reads from a snapshot cached with 5-second TTL in each app server's local process cache. Push leaderboard updates via SSE to subscribed users rather than polling (eliminates 90% of reads)
>
> [!TIP]
> **🟢 Staff+** — the leaderboard sorted set is a hot key — all reads go to one Redis shard. For a contest with 100K participants this is manageable; for a global contest with 1M: shard the leaderboard by rank range (top 100 is served from one shard, ranks 100-1000 from another) and merge at the API layer. Alternatively: serve approximate leaderboards (top 10% exact, rest approximate from a lower-frequency snapshot) — users care most about top positions
_Why the deep dives connect to the scaling problem: "Safe execution farm plus contest traffic spikes." Each deep dive addresses one dimension._
Interview script
1. Sandbox-first script.
2. "Clarifying questions: are we designing for a general OJ — all languages — or focused on a core set? And is the main challenge the execution security, the contest traffic spike, or both?"
3. "Good — all major languages, both challenges matter. Core features: submit code, execute safely, return verdict, contest leaderboard. Out of scope: code editor features, plagiarism detection."
4. "The unusual constraint here: user-submitted code is untrusted. The sandbox design is more interesting than the queue design. I'd lead with that."
5. "Sandbox: Docker container with seccomp profile — block socket syscalls (no network), hard cgroups for CPU time and memory, OOM kill at container level, read-only filesystem, PID limit against fork bombs. Pre-warm a container pool to eliminate cold-start latency."
6. "Queue: SQS between Submit API and worker pool. Submit API acknowledges immediately with a submission_id. Workers execute and write verdict to Redis (TTL 1hr). Client polls GET /submissions/:id."
7. "Per-language queues: Python runs 5-10× slower than C++. Mixed queue starves C++ users. Separate pools per language, sized by submission volume. Autoscale each pool independently by queue depth."
8. "Contest spikes: predictable traffic — pre-scale 30 minutes before start. Contest leaderboard in Redis sorted set with SSE push. Never poll leaderboard during a contest — too much QPS on one key."
Whiteboard
```
+-------------------+
| Web / Mobile |
| Client |
+---------+---------+
|
GET problems, submit code, poll
|
+---------v---------+
| API Server |
| auth from JWT |
| problem APIs |
| submit API |
| leaderboard API |
+----+---------+-----+
| |
read problems| |read submission status
| |
+---------v--+ +--v----------------+
| Problems DB | | Submissions DB |
| DynamoDB | | results, code, |
| problems, | | passed, metadata |
| test cases, | +---------+---------+
| code stubs | |
+------------ + |
|
enqueue job |
|
+---------v---------+
| Job Queue |
| SQS or similar |
+---------+---------+
|
pull job
|
+---------v---------+
| Submission Worker |
| picks runtime |
| loads problem |
| runs test harness |
+----+----------+----+
| |
execute code | | update leaderboard
| |
+---------------------v--+ +--v------------------+
| Sandboxed Containers | | Redis Sorted Set |
| python, java, js, etc | | competition ranks |
| CPU and memory limits | | fast top N reads |
| no network | +---------+-----------+
| timeout enforced | |
+-----------+------------+ |
| |
stdout or result |
| |
+-------------+-------------+
|
+--------v--------+
| API Server |
| returns status |
| to polling |
+--------+--------+
|
+--------v--------+
| Client |
| shows result |
| polls leaderboard|
+-----------------+
```
If you want the interview version, I would draw the simpler version first. Start with Client, API Server, Problems DB, Submissions DB, Queue, Worker, Sandboxed Containers, and Redis for leaderboard. Then explain that problem reads are synchronous, but submission execution is asynchronous because code runs for seconds and needs isolation.
The main idea is simple. Reads go straight through the API. Code submission goes through a queue to workers, workers execute inside locked down containers, results are stored in the submissions database, and leaderboard reads come from Redis instead of recomputing from the database every time.
Redis LuaToken BucketSliding Window CounterFail open vs closed
Every request hits the API gateway which runs a Lua script in Redis — atomically checks the counter and decrements it in a single operation, eliminating race conditions. Key = userId:window_id. Token Bucket: tokens refill at rate R, each request costs 1.
Problem
A rate limiter stops one user, bot, or client from sending too many requests in a short time.
The hard parts: distributed enforcement across many app servers, atomic check+decrement without race conditions, and choosing the right algorithm.
Failures
**Redis goes down — all counter state lost**
Two choices: fail open (allow all requests, lose rate limiting) or fail closed (deny all, break the product). Both are bad.
_Fix:_ Explicit choice: fail open for user-facing APIs (availability > protection), fail closed for auth endpoints (security > availability). Redis Sentinel for HA. Local in-process fallback limiter for fail-open path.
**Hot user / abusive IP hammers one Redis shard**
One shard gets 100× the write load. Latency spikes for all rate-limited requests on that shard.
_Fix:_ Consistent hash distributes users across shards. For known hot keys (viral API key, DDoS source IP), local in-process counter absorbs most checks. Redis is the shared truth, local cache is the fast path.
**Race condition: two servers check quota simultaneously and both allow a request that should be denied**
User gets 2× their quota allowance — rate limiter is ineffective.
_Fix:_ Lua script in Redis: INCR + TTL set + check, all atomic. Never do GET → check → SET as separate operations. Single-threaded Redis + Lua = no race possible.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 10K API servers, 100K requests/s total, each request does 1 Redis check, limit per user = 1000 req/min |
| Read QPS | 100K rate limit checks/s — all go to Redis |
| Write QPS | 100K INCR operations/s — Redis handles ~500K ops/s on one node, fine for one shard |
| Storage | Active users × 1 counter × 8 bytes × 2 windows = trivial. 10M users × 16 bytes = 160 MB — fits in single Redis node RAM |
| Cache math | Local in-process cache: each of 10K servers caches hot user counters. Reduces Redis QPS by ~80% for hot users. Redis sees only cache misses and periodic sync. |
| Verdict | Single Redis node handles 100K checks/s comfortably. Scale by sharding when users exceed 100M active keys. |
Design decisions
**Token bucket vs. sliding window vs. fixed window**
→ Sliding window counter
Fixed window allows burst at boundary (double rate in 2s straddling window). Token bucket is accurate but harder to implement distributed. Sliding window: curr_count + prev_count × (1 - elapsed%) is accurate and simple.
_Revisit when:_ Token bucket if smooth burst tolerance is required (API that should allow short bursts).
**Rate limit at gateway vs. at service level**
→ API gateway layer
Single enforcement point. No need to add rate limiting logic to every service. Rejected requests never reach service layer — saves service resources too.
_Revisit when:_ Service-level limits needed for internal service-to-service calls that bypass the public gateway.
**Per-user vs. per-IP vs. per-API-key**
→ Per-API-key primary, per-IP secondary
Authenticated requests use API key (accounts for shared IPs like NAT gateways). Unauthenticated requests fall back to IP. This prevents one mobile carrier's shared IP from affecting millions of legitimate users.
_Revisit when:_ Add per-endpoint limits for expensive operations (e.g., batch endpoints = 10x cost multiplier).
Follow-up Q&A
**How do you handle distributed Redis — do you need strong consistency across nodes?**
No. Slight over-allowance (10-20%) is acceptable for most rate limiters. Each shard holds counts for its key range. Race condition within a shard is eliminated by Lua. Race between shards is acceptable — user might get 110% of their quota in a rare race, not 1000%.
**What's your strategy for a DDoS with millions of unique IPs?**
Per-IP rate limiting at L7 (Nginx/HAProxy) or L3 (BGP blackholing for volumetric). This rate limiter is for application-layer limiting, not DDoS mitigation. Those are different systems. Always mention this distinction.
**How do you allow burst traffic (e.g., a user can burst to 2× for 10 seconds)?**
Token bucket with burst capacity: bucket size = sustained_rate × burst_window. User can drain the bucket instantly (burst), then refills at sustained_rate. This is what leaky bucket / token bucket is designed for — sliding window counter is harder to express burst with.
**How do you return accurate Retry-After headers?**
After INCR, read TTL on the window key. Return Retry-After: TTL seconds. For token bucket: (tokens_needed - tokens_available) / refill_rate = seconds to wait.
**How would you add per-plan rate limiting (free vs. paid tiers)?**
Store rate limit config per API key in a config service. Rate limiter fetches limit for the key (cached in local memory, refreshed every 60s). No code change needed to upgrade a customer's limits.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: throughput, queue lag, cache effectiveness. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: async replication lag <30s; failover promotes read replica. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — In-process token bucket per API server. Simple, no coordination. Problem: each server has its own quota — user can get N × (num_servers) requests by hitting all servers.
**v2 — Distributed** — Redis Lua script for atomic sliding window. Single enforcement point at API gateway. Handles millions of users correctly.
**v3 — Optimized** — Local in-process cache reduces Redis QPS by 80%. Per-plan configurable limits. Graduated limits for burst. DDoS-specific L3/L7 filtering as separate concern.
Why it's hard to scale
The hard part is that a rate limiter turns every incoming request into a fast, shared counter update. At small scale that sounds simple. At large scale, millions of requests all need low latency decisions, and those decisions must be consistent enough that one user cannot bypass limits just by hitting different servers.
There are three pain points you should call out. First, the state is write heavy. Every request updates a token bucket or counter, so your shared store can become the bottleneck. Second, correctness gets tricky under concurrency. Two servers can read the same remaining quota at once and both allow the request unless the whole read modify write step is atomic. Third, distribution makes it harder. If you shard by user or IP, you need all requests for that client to land on the same shard, and hot users or abusive IPs can overload one shard.
A good interview summary is this. Rate limiting is hard to scale because it needs very fast per request decisions, shared mutable state across many servers, and enough coordination to avoid race conditions without adding too much latency.
Key points
- **Enforce at gateway** — Single enforcement point at the API gateway layer.
- **Redis for shared state** — All app servers need to see the same counter.
- **Lua script = atomic** — Check + decrement in a single Lua script. No race condition.
- **Token bucket default** — Tokens refill at rate R, each request costs 1. Best default for most APIs.
- **Fail behavior** — If Redis is down: fail open (allow all) or fail closed (deny all). Name this explicitly.
> Mental model: Redis Lua script enforces atomically. Token bucket smooths bursts. Fail open vs closed is the hidden design choice.
Tradeoffs
**Fixed window vs sliding window** — Fixed window has boundary burst risk. Sliding window is accurate but uses more memory.
**Token bucket vs sliding window** — Token bucket smooths bursts best. Sliding window is more accurate for rate enforcement.
**Fail open vs fail closed** — Fail open = allow all when Redis is down. Fail closed = deny all. No right answer — depends on use case.
> "Fixed window is simple, sliding window is accurate, token bucket smooths bursts. Choose based on your traffic pattern."
Deep dives
#### Deep dive 1: Algorithm selection — token bucket vs. sliding window vs. fixed window
> [!CAUTION]
> **🔴 Weak** — use fixed window counting — INCR a key, expire at the window boundary
>
> [!WARNING]
> **🟡 Strong** — the algorithm choice has real operational tradeoffs, not just theoretical ones. Fixed window: simplest (INCR key, expire at window boundary), but the "double spend" attack — a burst of N requests at window T-1 and N requests at window T+1 = 2N requests in a 2-second window while the limit is N per window. Sliding window counter: curr_window_count + (prev_window_count × fraction_of_prev_window_elapsed) ≈ accurate sliding window at low memory cost. Token bucket: refill rate R tokens per second, cost 1 per request, bucket size B allows bursts up to B. Leaky bucket: queue requests and process at fixed rate — smoothest, but adds latency
>
> [!TIP]
> **🟢 Staff+** — sliding window counter for most API rate limiting (accurate, cheap). Token bucket for APIs that explicitly want to allow short bursts (e.g., batch endpoints). Fixed window only for very simple use cases where boundary bursts are acceptable. The algorithm choice should be justified against your NFRs — don't just name it
#### Deep dive 2: Distributed atomicity — why Lua scripts are mandatory
_The race condition: two servers both read the same counter (value: 999), both check 999 < 1000 (pass), both increment to 1000. User gets 2× their quota. The naive GET → check → INCR is broken_
> [!CAUTION]
> **🔴 Weak** — use Redis transactions (MULTI/EXEC)
>
> [!WARNING]
> **🟡 Strong** — Redis Lua scripts — the script executes atomically on the Redis server; no other command can run between any two lines of the Lua script. The script: (1) LOLWUT or EXISTS to check the key, (2) GET the current count, (3) if count < limit, INCR and return allowed, else return denied. All as one atomic operation
>
> [!TIP]
> **🟢 Staff+** — why not MULTI/EXEC? MULTI/EXEC is optimistic — if another client modifies the key between WATCH and EXEC, the transaction fails and must retry. Under high concurrency this creates a hot retry loop. Lua is unconditional — no retry needed. INCR alone is atomic too, but only for the increment — the conditional check requires Lua
#### Deep dive 3: Failure modes and degradation strategy
> [!CAUTION]
> **🔴 Weak** — fail closed — return 429 to all requests when Redis is down
>
> [!WARNING]
> **🟡 Strong** — what happens when Redis is unavailable? This is the question that separates senior from staff. Options: (1) fail open — allow all requests. Better for user-facing APIs where availability > protection. (2) Fail closed — deny all requests with 429. Better for auth endpoints where one compromised request is worse than a temporary outage. (3) Local in-process fallback — each app server maintains its own token bucket. Allows N × (num_servers) total requests instead of N, but prevents total failure
>
> [!TIP]
> **🟢 Staff+** — design: the failure mode should be a configuration option, not a code decision. Different endpoints have different failure mode requirements. The rate limiter must be explicit about which mode it's in and expose this via metrics. Additionally: Redis Sentinel for HA (automatic failover in <30s); circuit breaker pattern at the rate limiter client so a slow Redis doesn't add latency to every API request
_Why the deep dives connect to the scaling problem: "Fast per-request shared counter decisions." Deep dive 1 solves algorithm correctness. Deep dive 2 solves distributed atomicity. Deep dive 3 solves failure handling._
Interview script
1. Algorithm-first script.
2. "Clarifying questions: are we rate limiting by user, by IP, or by API key? And is the limit per-second, per-minute, or per-day? And what's the failure mode requirement — should the limiter fail open or closed?"
3. "Good — by API key, per-minute window. Failure mode: fail open for user-facing APIs, fail closed for auth endpoints. I'd make this a configuration option, not a code decision."
4. "Algorithm: sliding window counter. Fixed window has a boundary burst problem — 2N requests possible in 2 seconds straddling a boundary. Sliding window: count = curr_window + prev_window × (1 - elapsed%). Accurate and cheap."
5. "Implementation: Redis Lua script. The check-and-increment must be atomic — GET + check + INCR as separate operations has a TOCTOU race. Lua executes atomically on the Redis server, no race possible."
6. "Deployment: at the API gateway layer. Single enforcement point. Rejected requests never reach the service — saves downstream resources too. Service-level limits are a separate concern for internal traffic."
7. "Local in-process cache: for hot API keys, cache the counter locally with a 100ms TTL. Reduces Redis QPS by ~80%. Accept slight over-allowance (10-20%) in exchange for lower latency and less Redis load."
8. "Return Retry-After header on 429: after INCR, read TTL on the window key. Return Retry-After: TTL seconds. This is a small detail that signals production awareness."
Whiteboard
```
+----------------------+
| Config Service |
| rules and limits |
+----------+-----------+
|
periodic sync
|
+---------+ HTTPS +------------v-------------+
| Clients | --------------> | API Gateway / LB Layer |
| users | | auth parse + rate limit |
| IPs | | check before app traffic |
+---------+ +------+---------+---------+
| |
allow request | | reject request
| |
| v
| +----------------------+
| | 429 Response Builder |
| | limit remaining reset|
| +----------------------+
|
v
+---------+----------+
| Backend Services |
| social media APIs |
+--------------------+
Inside the gateway rate limiter path
extract client key
userId or IP or apiKey + endpoint rule
|
v
+----------+-----------+
| Shard Router |
| hash client key |
+----------+-----------+
|
v
+----------+-----------------------------------+
| Redis Cluster |
| shared bucket state across gateway instances |
| |
| shard 1 shard 2 shard 3 ... |
| +--------+ +--------+ +--------+ |
| |alice | |bob | |carol | |
| |tokens | |tokens | |tokens | |
| |refill | |refill | |refill | |
| +--------+ +--------+ +--------+ |
+-------------------+--------------------------+
|
atomic Lua script
|
v
read bucket -> refill tokens -> consume 1 -> return decision
Per shard HA
+------------------+
| Redis Primary |
+--------+---------+
|
replicate
|
+--------v---------+
| Redis Replica |
+------------------+
Request flow
```
FB Live Comments Medium
Fan-out at scale · Batching · Comment sampling
SSEKafkaCassandraBatching 100msComment sampling
Comments → Kafka partitioned by video_id. Delivery servers consume their partition and hold SSE connections. Comments are batched in 100ms windows — never pushed individually to 1M connections.
Problem
A real-time fan conversation around a live video with potentially 1M+ simultaneous viewers.
The hard part: naive per-comment fan-out to 1M SSE connections would collapse the system.
Failures
**A delivery server handling a mega-stream crashes**
All viewers connected to that server instantly lose their comment stream.
_Fix:_ Client auto-reconnects immediately. On reconnect, client sends last_seen_comment_id. Server replays missed comments from Cassandra. Recovery < 5 seconds.
**Kafka consumer lag grows during viral stream**
Comments appear with 30+ second delay. Real-time feel is broken.
_Fix:_ Monitor consumer lag per partition per video_id. Auto-scale delivery servers when lag exceeds 2s. Kafka partition by video_id ensures ordering within a stream is preserved on scale-out.
**Hot comment creates notification storm (1M replies in 10 min)**
Notification service is overwhelmed by reply fan-out.
_Fix:_ Notification batching: group replies from same comment, send one summary notification rather than 1M individual ones. Rate limit notifications per user (max 10/min). This is separate from the comment delivery pipeline.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 1M viewers on a viral stream, 1K commenters/s, avg comment = 100 bytes |
| Read QPS | 1M viewers × 1 update/100ms = 10M SSE messages/s — this is why batching is mandatory |
| Write QPS | 1K comments/s → after 100ms batching → 10 batch pushes/s to each delivery server serving 10K viewers |
| Storage | 1K comments/s × 100 bytes × 86400 × 365 ≈ 3 TB/year — Cassandra handles this |
| Cache math | Batch window: 100ms collects 100 comments per batch. 1M viewers receive a 100-comment batch vs. 100 individual pushes. 100× fan-out reduction. |
| Verdict | 100ms batching reduces effective fan-out by 100×. Without it, 10M SSE pushes/s is impossible. With it: 100K batch pushes/s — manageable. |
Design decisions
**SSE vs. WebSocket for delivery**
→ SSE (Server-Sent Events)
Comments flow one direction: server → viewer. WebSocket is bidirectional — overkill for this. SSE is simpler, HTTP/2 multiplexed, reconnects automatically, no upgrade handshake.
_Revisit when:_ WebSocket if viewers can react/reply inline (bidirectional interaction). SSE for pure delivery.
**Kafka partition by video_id vs. random partition**
→ Partition by video_id
All comments for a video go to one partition → consumed by one delivery server group → that server holds all viewer connections for that video. No cross-server coordination needed for fan-out.
_Revisit when:_ For mega-streams: multiple partitions per video_id with a coordinator that aggregates before fan-out.
**Show all comments vs. sampling at scale**
→ Full delivery up to 100K viewers, sampling above
Humans can't read > 10 comments/s anyway. For streams with > 100K viewers, sampled delivery (show 10% of comments) is indistinguishable from full delivery in perceived experience. Dramatically reduces fan-out load.
_Revisit when:_ Tiered sampling: VIP comments (verified accounts, top engagement) always shown, rest sampled.
Follow-up Q&A
**How do you ensure comment ordering across multiple commenters?**
Kafka partition key = video_id ensures all comments for a stream are ordered within the partition (Kafka offset order). Delivery server appends a server-side sequence number per stream. Client renders in sequence order.
**What happens when a user joins mid-stream?**
On connect, client requests last N comments (e.g., last 30) from Cassandra. Delivery server responds with comment history, then transitions to live SSE stream. Client deduplicates if there's overlap between history and live stream using comment_id.
**How do you handle comment moderation at 1K comments/s?**
Async ML classifier on the write path: comment → Kafka → classifier → store or soft-block. Fast path: store comment immediately (latency matters for live). Slow path: classifier removes blocked comments within 5s. Users may briefly see blocked comments.
**How does this change for a 50M viewer stream (Super Bowl)?**
Two changes: (1) sampling becomes aggressive (0.1% of comments shown — still 1 comment/s per viewer at 1K comments/s). (2) Dedicated comment delivery cluster for the stream, pre-scaled, isolated from normal traffic.
**How do you handle reconnections without missing comments?**
Client stores last_seen_comment_id. On reconnect, includes it in request. Server replays comments after that ID from Cassandra, then transitions to live SSE. Cassandra query: SELECT * FROM comments WHERE video_id=? AND id > last_seen ORDER BY id LIMIT 50.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: throughput, queue lag, cache effectiveness. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: async replication lag <30s; failover promotes read replica. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Long-polling every 2 seconds. Simple DB for comments. Works up to 10K viewers.
**v2 — Real-time** — SSE delivery. Kafka partitioned by video_id. Cassandra for comment storage. 100ms batching. Handles 1M viewers.
**v3 — Mega-streams** — Adaptive comment sampling. Dedicated stream clusters. Tiered delivery (VIP comments always, rest sampled). ML moderation pipeline.
Why it's hard to scale
The hard part is fan-out under extreme skew. One new comment is a tiny write, but it may need to reach thousands or millions of viewers for the same live video almost immediately.
That creates three scaling pain points. First, the read side explodes because every active viewer needs a near real-time stream, so polling falls over and even push connections become expensive at big scale. Second, hot videos create uneven load. Most streams are quiet, but one viral stream can concentrate huge traffic, connection count, and comment throughput onto a small part of the system. Third, once viewers for the same video are spread across many realtime servers, you need coordination so every server knows which new comments to forward.
The extra twist is that the best design changes for mega-streams. For normal videos, SSE plus pub sub works well. For massive streams, you usually stop trying to show every comment and switch to sampling or snapshot style delivery because humans cannot read thousands of comments per second anyway.
Key points
- **Scope it first** — Core: post comment, deliver to all live viewers in near-real-time, show recent comment history on join. Out of scope: reactions, moderation pipeline, comment replies.
- **Batch — the single most important decision** — 100ms coalescing window. At 1M viewers and 1K comments/sec, per-comment push = 1B SSE pushes/sec. Batching reduces this to 10M/sec. This is a requirement, not an optimization.
- **Kafka partition = delivery server locality** — Partition key = video_id. One partition consumed by one delivery server group. All viewer connections for a video are on those servers. Zero cross-server fan-out coordination needed.
- **SSE over WebSocket** — Comments flow one direction: server → viewer. SSE is simpler (standard HTTP/2, auto-reconnect, no upgrade). Only use WebSocket if viewers need to send data inline.
- **Late joiner catch-up** — On connect, client sends last_seen_comment_id. Server queries Cassandra for comments after that ID (LIMIT 30), then switches to live SSE. Client deduplicates by comment_id.
- **Sampling at extreme scale** — 50M viewers (Super Bowl): show 0.1% of comments — still 1 comment/sec per viewer at 1K comments/sec. Humans cannot read faster. VIP comments (verified accounts) always shown regardless of sampling rate.
- **Failure mode to name** — Delivery server crash: client auto-reconnects, sends last_seen_comment_id, replays missed comments from Cassandra. Cassandra is the durable fallback — never rely on SSE delivery alone for correctness.
> The key insight: batching is everything. 1M individual pushes = system collapse.
Tradeoffs
**Per-comment push vs batching** — Per-comment is low-latency but collapses at 1M viewers. 100ms batching is the only viable approach at scale.
**SSE vs WebSocket** — SSE is simpler for one-directional server push. WebSocket needed if viewers also react inline. For comment delivery only, SSE is the right default.
**Show all comments vs sampling** — At 1M viewers and 1K comments/sec, full delivery means 1B SSE messages/sec. Sampling (show 10% of comments at extreme scale) is indistinguishable to humans — we cannot read faster than ~10/sec anyway.
**Kafka partition by video_id vs random** — Partition by video_id ensures all comments for one stream go to one partition consumed by one delivery server group. This eliminates cross-server fan-out coordination. Random partition would require cross-server routing on every delivery.
> "The key scaling insight is batching. Kafka partition by video_id gives delivery locality. Sample at extreme scale."
Deep dives
#### Deep dive 1: Batching — the mandatory optimization that makes fan-out viable
_1M viewers × 1 comment per 100ms = 10M SSE pushes per second if done naively. No system survives this. The critical insight: batching is not an optimization, it's a requirement_
> [!CAUTION]
> **🔴 Weak** — batch at the delivery server
>
> [!WARNING]
> **🟡 Strong** — batch at the Kafka consumption layer. Delivery server consumes Kafka for a video_id partition, buffers incoming comment events for 100ms, then pushes a single batch payload to all 10K connected clients. The batch payload: {comments: [{id, text, author, timestamp}, ...], viewer_count: 1000000}
>
> [!TIP]
> **🟢 Staff+** — the batching window is a tunable parameter. 100ms gives 10 batches/second — fast enough for a live conversation feel, slow enough to be viable at scale. For ultra-high-volume streams: adaptive batching — increase window size as comment rate increases, keeping per-viewer bandwidth constant regardless of comment volume. The coalescing also improves compression ratio on the SSE payload (repeated fields in multiple comments compress well)
#### Deep dive 2: Partitioning and delivery server locality — eliminating cross-server coordination
> [!CAUTION]
> **🔴 Weak** — route comment events to any available delivery server, then fan-out via cross-server pub/sub
>
> [!WARNING]
> **🟡 Strong** — Kafka partition key = video_id. All comments for a video go to one Kafka partition. One delivery server group (2-3 servers) consumes each partition. By contract, all viewers of a video are connected to servers in the group that consumes that video's partition. Result: fan-out requires zero cross-server communication — the consuming server directly has all viewer connections
>
> [!TIP]
> **🟢 Staff+** — what about load balancing? A viral video creates a hot partition (1M viewers on 2-3 servers). Mitigation: hot video detection — when a video exceeds a viewer threshold, allocate multiple Kafka partitions for it and a larger server group. Connection routing: clients are directed to servers in the right group via DNS-based load balancing with video_id affinity. For mega-streams (Super Bowl: 50M+ viewers): dedicated server cluster for the stream, isolated from regular traffic, pre-provisioned
#### Deep dive 3: Historical comments and late joiners — the catch-up problem
> [!CAUTION]
> **🔴 Weak** — send the entire comment history on connect, then switch to live stream
>
> [!WARNING]
> **🟡 Strong** — a viewer joins a live stream 30 minutes in. They need: (1) the last N comments to provide context, (2) then a seamless transition to the live comment stream. Weak answer: load from DB then connect to SSE. Strong answer: explicit catch-up protocol. On SSE connection: client sends last_comment_id = 0 (new viewer). Server queries Cassandra: SELECT * FROM comments WHERE video_id=? AND id > 0 ORDER BY id ASC LIMIT 30. Returns the 30 most recent comments. Then switches client to live SSE stream. Client deduplicates by comment_id in case the SSE stream delivers a comment that was already in the catch-up response
>
> [!TIP]
> **🟢 Staff+** — the catch-up query uses a cursor (comment_id) not a timestamp — comment_ids are monotonically increasing Snowflake IDs that encode time, so they're both ordered and unique. The transition from catch-up to live is seamless: client tracks max comment_id received, SSE stream starts from next ID
_Why the deep dives connect to the scaling problem: "Fan-out under extreme skew." Deep dive 1 solves the fan-out rate problem. Deep dive 2 solves the routing problem. Deep dive 3 solves the late-joiner problem._
Interview script
1. Batching-first script.
2. "Clarifying questions: are we delivering comments to live video viewers only, or also recorded videos? And what's the scale target — Twitch-size (thousands of viewers) or Facebook Live scale (millions)?"
3. "Good — Facebook Live scale, millions of viewers. Core features: submit comment, deliver to all viewers in near-real-time, show comment history on join. Out of scope: reactions, moderation pipeline."
4. "The single most important design decision: batching. At 1M viewers and 1K comments/sec, per-comment push = 1B SSE messages/sec. 100ms batching: push the last N comments as a batch payload, 10 times/second. Reduces to 10M messages/sec. This is a requirement, not an optimization."
5. "Architecture: Comment API → Kafka (partition by video_id) → Delivery servers (consume their partition) → SSE to viewers. Kafka partition key = video_id ensures all comments for one stream hit one delivery server group. Zero cross-server coordination for fan-out."
6. "Comment storage: Cassandra, partitioned by video_id. On viewer connect: send last 30 comments from Cassandra (catch-up), then switch to live SSE stream. Client deduplicates by comment_id at the boundary."
7. "For a 50M viewer stream: aggressive sampling — show 0.1% of comments. Humans cannot read faster than ~10/sec. VIP and high-engagement comments always shown. This makes the system feasible at Super Bowl scale."
8. "Key tradeoff: SSE over WebSocket. Comments are unidirectional — server pushes, viewers watch. SSE is simpler, HTTP/2 multiplexed, auto-reconnects. WebSocket adds complexity we don't need for delivery-only."
Whiteboard
```
+-------------------+
| Commenter App |
| POST comment |
+---------+---------+
|
v
+---------------------+
| API / Comment |
| Management Service |
+----+------------+---+
| |
write comment| | publish event
v v
+----------------+ +-------------------+
| Comments DB | | Pub/Sub Bus |
| DynamoDB | | Redis or similar |
+----------------+ +---------+---------+
|
fan out to interested servers
|
+-------------------------+-------------------------+
| | |
v v v
+---------------+ +---------------+ +---------------+
| Realtime Srv 1| | Realtime Srv 2| ... | Realtime Srv N|
| SSE conns | | SSE conns | | SSE conns |
| local map | | local map | | local map |
+------+--------+ +------+--------+ +------+--------+
| | |
v v v
+-------------+ +-------------+ +-------------+
| Viewer Apps | | Viewer Apps | | Viewer Apps |
| SSE stream | | SSE stream | | SSE stream |
+-------------+ +-------------+ +-------------+
History and catch-up path
Viewer App
|
| GET /comments/:liveVideoId?cursor=lastCommentId&pageSize=10
v
+-------------------+
| Comment Management |
| Service |
+---------+---------+
|
v
+-------------------+
| Comments DB |
| paginated reads |
+-------------------+
```
If you want the best interview version, I would say it out loud like this. Comments are written through a comment service into DynamoDB. New comment events are published to a pub sub layer. Realtime servers hold SSE connections to viewers and push comments out. Historical comments and reconnect catch-up come from the database using cursor pagination.
For scale, you'll want one extra note on the diagram. Put a load balancer in front of the realtime servers and say you try to co-locate viewers of the same liveVideoId on the same server or small set of servers to reduce fanout waste.
FB Post Search Medium
Inverted index · Privacy filtering · Near-real-time index
Posts → Kafka → async index worker tokenizes and writes to Elasticsearch with a privacy_level field. On search, ES handles text matching. A post-query app layer filter enforces friend-list visibility — friend lists are too large and dynamic for ES.
Problem
Keyword search over billions of posts at Facebook scale.
You can't scan raw post text at request time — the corpus is enormous. The real problem is precomputing an inverted index and keeping it fresh.
Failures
**ES index lag during a viral event**
Posts about breaking news don't appear in search for 5+ minutes. Users see stale results.
_Fix:_ Dedicated fast-lane Kafka topic for high-engagement posts (ML classifier identifies potential viral content within seconds of publish). Fast-lane ES indexer with smaller batch size (<1s latency). Normal posts use standard pipeline.
**Like-count index update storms a hot term**
Popular keyword has millions of posts. One viral post gets 1M likes in an hour → 1M index updates to the same ES shard.
_Fix:_ Batch like updates: count likes in Redis, flush to ES every 5 minutes. Like-sorted index has approximate freshness — acceptable. Don't update ES on every individual like.
**Privacy filter in app layer creates N+1 query problem**
Search returns 100 results. App layer checks each post's privacy settings + friend graph. 100 DB queries per search request.
_Fix:_ Embed privacy_level in ES document. App layer only needs friend-graph check for 'friends only' posts. Batch friend-graph check in one query. For public posts: zero DB calls.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 3B users, 500M posts/day, 100M searches/day, avg 10 likes/post |
| Read QPS | 100M searches / 86400 ≈ 1,157 search QPS — manageable for ES cluster |
| Write QPS | 500M posts / 86400 ≈ 5,787 post index writes/s + 500M × 10 likes / 86400 = 57,870 like events/s → batched to ~200 index updates/s |
| Storage | 3B posts × avg 500 chars × 2 bytes ≈ 3 TB raw text. ES index ≈ 2× = 6 TB. Inverted index per term over 3B docs. |
| Cache math | Hot search terms: top 10K queries account for ~60% of search traffic. Cache results for 30s: 10K × 10KB ≈ 100 MB Redis cache absorbs 60% of ES load. |
| Verdict | Like-update write amplification is the hidden bottleneck. 58K like events/s → 200 batched index updates/s is the key optimization. |
Design decisions
**Sort by recency vs. sort by popularity (likes)**
→ Two separate indexes, query routing by sort parameter
Recency sort: posts indexed by created_at (append-only, no updates needed). Popularity sort: posts ranked by like_count (frequent updates). Different update patterns require different optimization strategies.
_Revisit when:_ Unified index with sort-at-query-time is simpler but requires ES to handle the like-update storm.
**Privacy filtering: ES vs. app layer**
→ Coarse filter in ES (privacy_level field), fine filter in app layer (friend graph)
Storing full friend lists in ES documents is impossible (500M users × avg 1K friends = 500B entries in ES). Hybrid: ES filters obviously inaccessible posts, app layer handles 'friends only' edge cases.
_Revisit when:_ Graph DB (Neo4j) as a friend-graph service that the app layer queries for batch privacy checks.
**Freshness SLA for new posts appearing in search**
→ < 1 minute for public posts, < 5 minutes for friends-only posts
Users expect to search for something they just read about and find it quickly. 1 minute is the minimum perceptible freshness degradation. Friends-only posts are less time-critical.
_Revisit when:_ Real-time indexing (<5s) for verified accounts and high-engagement posts (breaking news).
Follow-up Q&A
**How do you handle multi-word queries vs. single keyword?**
ES boolean query: AND for exact phrase match (higher score), OR for any-term match (lower score). Phrase proximity boosting: 'coffee shop' as adjacent words scores higher than same words far apart. ES handles this natively with match_phrase and match queries.
**How would you add autocomplete to the search box?**
Separate ES index with completion suggester field. Stores prefix trees of common search terms. Query: prefix match on first few typed characters. Response time < 50ms. Different index from the main post search index.
**How do you prevent search from surfacing harmful content?**
Content moderation pipeline runs before indexing. Posts flagged by ML classifier are soft-blocked (stored but not indexed). User-level block list: exclude posts from blocked users at query time (ES filter). Trending topics monitoring for coordinated abuse.
**How do you handle search in 100 languages?**
Per-language ES index (or per-language field in one index with language-specific analyzers). Language detection on post at index time. Query routed to language-appropriate analyzer. Transliteration index for cross-script queries (Hindi typed in Roman script).
**What's the latency budget for a search request?**
Target: P99 < 500ms. Budget: ES query 100ms + privacy filter 50ms + result hydration 50ms + network 100ms = 300ms. Remaining 200ms for cache miss + query parsing. Achieve with: query result caching (Redis, 30s TTL), connection pooling to ES, async result hydration.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: throughput, queue lag, cache effectiveness. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: async replication lag <30s; failover promotes read replica. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — LIKE '%query%' on PG posts table. Works for 1M posts. Completely breaks at 100M posts.
**v2 — Inverted index** — Elasticsearch for post text + recency index. Kafka async indexing. Privacy coarse filter in ES. Basic friend-graph check in app layer. Handles billions of posts.
**v3 — Scale + freshness** — Batched like-count updates. Fast-lane indexing for viral posts. Per-language indexes. Autocomplete service. ML ranking layer. Result caching.
Why it's hard to scale
The hard part in FB Post Search is that search looks read heavy, but the indexing work is actually very write heavy. Every new post fans out into many index updates, and likes can create even more updates if you sort by popularity.
There are three scaling pain points to call out. First, you cannot scan raw posts at query time because the data is far too large, so you need an inverted index that maps terms to post IDs. Second, some terms are extremely hot, which means their posting lists get huge and expensive to store, sort, and query. Third, freshness matters. New posts should appear quickly, so you need fast ingestion and index updates without overwhelming the system.
A fourth issue is sorting. Recency is manageable, but sorting by like count is much harder because likes change constantly. That means one user action can force many index updates unless you use approximation or batching.
So the short interview answer is this. FB Post Search is hard because it combines massive inverted indexes, hot keywords, heavy write amplification from posts and likes, and a tight freshness requirement.
Key points
- **Inverted index** — Each keyword points to post IDs that contain it. Never scan raw posts at search time.
- **Write path** — Post → Kafka → index worker tokenizes → updates inverted index in ES.
- **Two sort indexes** — One ordered by creation time (recency). One ordered by like count (popularity).
- **Privacy in app layer** — Post-query filter. Friend lists are too large and dynamic for ES.
- **Shard by keyword** — So writes and reads spread across many machines.
> Mental model: precompute search with an inverted index. Write path tokenizes posts. Read path serves from indexes with caching.
Tradeoffs
**Fast reads vs write cost** — Precomputed indexes make search fast but every post creates many index writes. Accept this — request-time scanning is too slow at billions of posts.
**Exact like ranking vs batched updates** — Updating every keyword ranking on every like is expensive at 57K likes/sec. Batched Redis → ES flush every 5 min keeps performance healthy at the cost of slight staleness.
**Privacy filter in ES vs app layer** — Storing full friend lists in ES documents is impossible at 3B users × avg 1K friends. Coarse privacy_level field in ES plus friend-graph check in app layer is the only viable hybrid.
**One unified index vs recency + popularity indexes** — A single index with sort-at-query-time couples update patterns. Recency index is append-only; popularity index needs frequent like-count updates. Separate indexes let each optimize independently.
> "FB Post Search is an inverted index system. Write path tokenizes posts and updates per-keyword indexes. Read path serves from those indexes with caching, sharding, and batching."
Deep dives
#### Deep dive 1: Inverted index design — term mapping to post IDs with two sort orders
> [!CAUTION]
> **🔴 Weak** — one unified Elasticsearch index, sort at query time
>
> [!WARNING]
> **🟡 Strong** — the core data structure: a map from each keyword to an ordered list of post IDs containing that keyword. Two sort orders are required: recency (post_id DESC or created_at DESC) and popularity (like_count DESC). Weak answer: one index, sort at query time. Strong answer: two separate indexes, each optimized for its sort order. The recency index is append-only (new posts are always the most recent — just append to the front). The like-count index is updated on every like — much more expensive because one like on a popular post triggers updates to potentially thousands of term lists
>
> [!TIP]
> **🟢 Staff+** — don't update the like-count index on every individual like. Batch like updates in Redis: increment a like counter in Redis, flush to the index every 5 minutes. This reduces write amplification from O(terms_per_post) per like to O(terms_per_post) per 5-minute window. The tradeoff: like-count rankings are slightly stale (up to 5 minutes) — acceptable, as users don't notice
#### Deep dive 2: Privacy filtering — the friend-graph problem
_Privacy is the hardest part of FB Post Search. Every result must be filtered against the viewer's permission to see it_
> [!CAUTION]
> **🔴 Weak** — post-query filter for every result
>
> [!WARNING]
> **🟡 Strong** — coarse filter in ES (privacy_level field: PUBLIC, FRIENDS, PRIVATE) combined with a friend-graph check in the app layer for FRIENDS-only posts
>
> [!TIP]
> **🟢 Staff+** — why not store friend lists in ES documents? The friend list for a user can be millions of people and changes constantly — storing it in every post document would make ES documents enormous and constantly out of date. The hybrid approach: ES returns candidates filtered by privacy_level. For FRIENDS posts in the result set, batch-query the social graph service: "Is viewer X friends with any of these [user_ids]?" This is a set intersection problem — solved efficiently with Bloom filters per user (is viewer in author's friend set?) or with a graph adjacency lookup. The friend check adds ~20ms to P99 latency — acceptable for a search query
#### Deep dive 3: Freshness — new posts appearing in search within 1 minute
> [!CAUTION]
> **🔴 Weak** — sync Elasticsearch from PostgreSQL with a scheduled batch job every 10 minutes
>
> [!WARNING]
> **🟡 Strong** — users expect to search for something they just posted and find it. The ingestion pipeline: post created in PG → Kafka event → index worker → ES update. End-to-end target: < 1 minute
>
> [!TIP]
> **🟢 Staff+** — bottlenecks to address: (1) ES bulk indexing batch size — larger batches are more efficient but add latency. At 5,787 posts/second, even 1-second batches give reasonable throughput. (2) ES refresh interval — by default ES refreshes the search index every 1 second (making indexed docs visible to search). For breaking news content: reduce refresh interval to 100ms for the first 5 minutes after a high-engagement post is created (dynamic refresh rate based on post engagement velocity). (3) High-engagement posts fast-lane: ML classifier identifies potentially viral posts within seconds of publish — route these to a priority Kafka topic with a dedicated low-latency indexer
_Why the deep dives connect to the scaling problem: "Massive inverted indexes, hot keywords, write amplification, and freshness." Each deep dive addresses one constraint._
Interview script
1. Two-path script.
2. "Clarifying questions: are we searching all posts or only the searcher's social graph? And what's the freshness requirement — how quickly should a new post appear in search?"
3. "Good — full search with privacy filtering, freshness target of under 1 minute for new posts. Core features: text search with relevance ranking, privacy-aware results. Out of scope: hashtags, people search."
4. "Two distinct paths with different concerns: write path (index new posts) and read path (serve search queries). I'd walk them separately."
5. "Write path: post created → Kafka async → index worker tokenizes and NLP-processes text → indexes in Elasticsearch with privacy_level field and timestamp. Target latency: post visible in search within 60 seconds."
6. "Read path: search query → ES compound query (text match + privacy_level filter) → app layer friend-graph check for FRIENDS posts → return results. Privacy filter in app layer, not ES, because friend lists are too large and dynamic to store in ES documents."
7. "Like-count ranking: don't update ES on every like at 57K likes/sec. Batch like counts in Redis, flush to ES every 5 minutes. Like-sorted results are slightly stale — acceptable for search, not for the post itself."
8. "Key tradeoff: one unified ES index vs separate recency and popularity indexes. Unified is simpler but couples different update patterns. Recency index is append-only. Popularity index needs frequent updates. Separate indexes let each optimize independently."
Whiteboard
```
+-------------------+
User Search Request -----> | CDN / Edge |
+---------+---------+
|
v
+-------------------+
| API Gateway |
| auth rate limit |
+---------+---------+
|
v
+-------------------+
| Search Service |
+----+----------+---+
| |
cache hit? | | fetch posts and fresh likes
| v
| +--------------+
| | Post Service |
| +--------------+
| +--------------+
| | Like Service |
| +--------------+
v
+-----------------------+
| Distributed Search |
| Cache TTL < 1 minute |
+-----------+-----------+
|
cache miss
|
v
+-------------------------------------------+
| Keyword Index Store |
| sharded by keyword |
| |
| creation index = list by recency |
| likes index = sorted set by like score |
+-------------------+-----------------------+
|
hot keywords | cold keywords
|
+---------------+---------------+
| |
v v
+-------------+ +-------------+
| Redis shard | | Blob store |
| in memory | | S3 or R2 |
+-------------+ +-------------+
Write path
==========
Post Create ----> Post Service ----+
|
Like Event -----> Like Service ----+----> Kafka or event log ----> Ingestion workers
|
v
+----------------------+
| Tokenizer |
| split into keywords |
| optional bigrams |
+----------+-----------+
|
+------------------------+----------------------+
| |
v v
+--------------------------+ +--------------------------+
| Update creation indexes | | Update likes indexes |
| add postId per keyword | | sorted set score updates |
+--------------------------+ +--------------------------+
|
v
+------------------------------+
| Optional like batcher or |
| approximate milestone writer |
+------------------------------+
```
The mental model is two pipelines. One pipeline builds keyword indexes from posts and likes. The other pipeline serves search by reading those indexes fast, usually from cache or Redis.
If you were drawing this in an interview, I would start with just User, API Gateway, Search Service, Ingestion Service, and Index Store. Then add cache, Kafka, like batching, and cold storage only if the interviewer pushes on scale or freshness trade-offs.
Search combines geo_distance filter and full-text match in a single Elasticsearch query. PostgreSQL is source of truth. Average ratings are maintained in Redis and synced to PG asynchronously. Photos live on S3 + CDN.
Problem
Local business discovery and trust. Find a place nearby, filter by category and location, decide based on reviews and ratings.
Two challenges: geospatial search at scale, and keeping ratings consistent.
Failures
**Elasticsearch index goes out of sync with PostgreSQL**
Search results show stale business info (wrong hours, closed businesses still appearing).
_Fix:_ CDC (Change Data Capture) from PG → Kafka → ES indexer. Monitor sync lag with alerting at >60s. Periodic full reconciliation job (nightly) to catch missed events.
**Hot neighborhood (NYC midtown) hammers ES with geo queries**
One ES shard handling all NYC queries gets overloaded.
_Fix:_ Shard ES index by geohash prefix — ensures queries for the same region go to the same shard (locality). Add replica shards for hot regions. Route to nearest replica.
**Review rating aggregation is slow under write load**
New review posted, but average rating on search result takes minutes to update.
_Fix:_ Maintain running average in Redis (HINCRBY sum + count). Async batch-sync to PG every 30s. ES rating field updated via Kafka event. Rating is eventually consistent — seconds of lag is acceptable.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 100M DAU, 10 searches/day, 5 page views/search, 1M new reviews/day |
| Read QPS | 100M × 10 / 86400 ≈ 11,600 search QPS — served by Elasticsearch |
| Write QPS | 1M reviews / 86400 ≈ 12 review write QPS — trivially small |
| Storage | 10M businesses × 5 KB metadata ≈ 50 GB PG. ES index ≈ 3× PG size = 150 GB. Reviews: 500M × 500 bytes ≈ 250 GB. |
| Cache math | Top 10K businesses get 80% of page views. Cache business pages: 10K × 50 KB ≈ 500 MB — tiny Redis cache handles this. |
| Verdict | Read-dominated by 1000:1. ES and CDN are the critical path. PG writes are trivial. |
Design decisions
**ES vs. PostgreSQL with PostGIS for search**
→ Elasticsearch with geo_distance + full-text
PG with PostGIS can do geo queries, but combining geo + full-text + category filters + rating sort in one PG query gets slow at 11K QPS. ES handles all four dimensions natively in one query.
_Revisit when:_ PG + PostGIS works fine up to ~1K QPS geo+text queries. Only switch to ES at higher scale.
**Running average in Redis vs. recompute from reviews table**
→ Precomputed running average in Redis
Recomputing avg(rating) across 10K reviews on every search request at 11K QPS = impossible. Maintain running sum and count: avg = sum/count. O(1) update per new review.
_Revisit when:_ Could do approximate aggregation with Redis HyperLogLog for distinct reviewer count.
**User photos: serve from S3 directly vs. CDN**
→ S3 + CDN with image resizing on first access
Business photos are requested millions of times. Serving from origin = massive S3 egress cost. CDN caches at edge. On-demand image resizing (Lambda@Edge or Imgix) serves appropriate resolution per device.
_Revisit when:_ Pre-generate thumbnails at upload time if resizing latency on first access is too high.
Follow-up Q&A
**How do you rank search results?**
Multi-factor score: ES relevance score (text match quality) × distance decay function × rating × review count. Tuned with A/B testing. ML re-ranking for personalized results (separate service, separate concern from retrieval).
**How do you handle fake reviews?**
ML classifier on review text + user behavior signals (new account, same IP, suspiciously similar text). Soft delete (hide from display, keep for model training). Business owner can flag. Manual review queue for borderline cases.
**How do you keep ES in sync when PG is the source of truth?**
Debezium CDC: streams PG WAL changes to Kafka. ES indexer consumes Kafka topic, applies updates to ES. Idempotent — replaying the same change twice is safe. Nightly reconciliation job for drift detection.
**What if a business has 100,000 reviews — is that a hot partition in PG?**
Reviews table partitioned by business_id + time. Reads are paginated — never full table scan. Business-level aggregates (avg rating, review count) are precomputed, not queried from the reviews table directly. PG handles this fine.
**How would you add 'open now' filtering?**
Store hours as structured data (day_of_week, open_time, close_time). At query time, compute current UTC + business timezone → is it open? Apply as a post-ES filter in app layer (too dynamic to index). Cache 'open now' status per business with 15-min TTL.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: throughput, queue lag, cache effectiveness. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: async replication lag <30s; failover promotes read replica. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — PG for everything. LIKE query for search. No geo filtering. Works for a city with 1K businesses and 1K users.
**v2 — Real search** — Elasticsearch for text + geo. PG stays as source of truth. CDC sync. Redis for rating cache. CDN for photos. Handles city-scale.
**v3 — Global scale** — Regional ES clusters. Sharding by geohash. ML ranking. Personalization. Business owner API. Real-time review moderation.
Why it's hard to scale
The hard part in Yelp is search, not writes. You need to search by text, category, and location at the same time, and location is especially tricky because geospatial queries do not scale well with a plain relational lookup.
There are three main pain points you should call out. First, search is multi-dimensional. A user might ask for coffee, in a certain area, in a certain category, so you need full-text indexing plus geospatial indexing, not just a normal database query. Second, reads dominate writes by a lot, so popular areas create heavy search and business page traffic and you need read scaling with replicas or caching. Third, some derived data like average rating must stay cheap to read, so you usually precompute it instead of recalculating from all reviews on every search.
A good interview summary is this. Yelp is hard because it combines read-heavy traffic with geospatial search and text search in one request. The write path for reviews is comparatively small, so the main challenge is making search fast while keeping results fresh enough.
Key points
- **PostgreSQL = source of truth** — Businesses, reviews, and users.
- **Elasticsearch = search index** — geo_distance filter + full-text in one query.
- **Redis = rating cache** — Average ratings served from Redis for fast reads. Synced to PG asynchronously.
- **Photos = S3 + CDN** — Never serve photos from app servers.
- **Search → ES, not PG** — Never do SQL LIKE queries on business names at scale.
> Three databases, one per concern. PG for writes and truth. ES for search. Redis for hot reads.
Tradeoffs
**ES geo_distance vs PostGIS** — ES handles both text and geo in one compound query. PostGIS is more powerful for complex spatial queries but adding a second query join adds latency and complexity.
**Redis for ratings vs recompute from DB** — Redis running average (sum + count) gives sub-millisecond rating reads. Recomputing avg(rating) at 11K QPS from a reviews table is O(N) per request — impossible.
**CDC sync vs dual-write for ES freshness** — Dual-write is simpler but ES and PG can diverge on partial failure. CDC (Debezium → Kafka → ES indexer) is more reliable because it catches all writes regardless of source, at the cost of eventual consistency (~5s lag).
**"Open now" filter in ES vs app layer** — Hours data changes every minute by definition (time passes). Indexing a boolean that changes constantly creates massive re-indexing overhead. Compute "open now" in app layer with 15-min TTL cache per business — far cheaper.
> "PostgreSQL for truth, ES for search (geo + text), Redis for hot reads. Three stores, each with one job."
Deep dives
#### Deep dive 1: Geospatial + text search — Elasticsearch as the single query surface
> [!CAUTION]
> **🔴 Weak** — run separate queries for geo (PostGIS) and text (LIKE), merge results in the app layer
>
> [!WARNING]
> **🟡 Strong** — the scaling pain is that a user query like "best sushi near me open now" combines full-text search, geo filtering, category filtering, and rating sorting in one request. Weak answer: separate queries to separate services, merge in app layer. Strong answer: Elasticsearch as a single query surface with a compound query: geo_distance filter (within X km of lat/lng) + multi_match on business name/category/description + term filter on category + range filter on rating + sort
>
> [!TIP]
> **🟢 Staff+** — the ES query shape matters for performance. Structured fields (category, rating, hours) should use filter context (cached, no scoring overhead) and text fields should use query context (scored, more expensive). The geo_distance filter is the most selective first — apply it first to reduce the candidate set before text scoring. For "open now": this is a computed field that changes every minute — too dynamic to index. Apply as a post-query filter in app layer, not in ES. Cache "is_open" status per business with 15-minute TTL
#### Deep dive 2: Keeping ES in sync with PostgreSQL — CDC and eventual consistency
_ES is derived data — PostgreSQL is the source of truth. Sync strategy options: (1) dual write (write to PG + ES in same request) — simple but PG and ES can diverge on partial failure; (2) CDC (Debezium reads PG WAL → Kafka → ES indexer) — eventual consistency, reliable, standard production approach; (3) async event-driven (after PG write succeeds, publish Kafka event → ES indexer) — same reliability as CDC but more explicit_
> [!CAUTION]
> **🔴 Weak** — Oversimplify keeping es in sync with postgresql — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — ES is derived data — PostgreSQL is the source of truth. Sync strategy options: (1) dual write (write to PG + ES in same request) — simple but PG and ES can diverge on partial failure; (2) CDC (Debezium reads PG WAL → Kafka → ES indexer) — eventual consistency, reliable, standard production approach; (3) async event-driven (after PG write succeeds, publish Kafka event → ES indexer) — same reliability as CDC but more explicit
>
> [!TIP]
> **🟢 Staff+** — CDC is the production default because it works for all write patterns (including bulk imports, migrations, and direct DB writes by other services) without requiring every writer to know about ES. Lag: typically 1-5 seconds — acceptable for search. Monitor lag metric: alert at >60s. For new business data (most time-sensitive): reduce batch size in the indexer for faster propagation
#### Deep dive 3: Rating aggregation — precomputed running average
_The access pattern for ratings: millions of reads per business page, one write per new review. Recomputing avg(rating) from all reviews on every page view is O(N) per request at 11,600 QPS — impossible_
> [!CAUTION]
> **🔴 Weak** — cache the result
>
> [!WARNING]
> **🟡 Strong** — maintain a running average in the database: (sum_of_ratings, review_count) per business. On new review: UPDATE businesses SET sum_of_ratings = sum_of_ratings + ?, review_count = review_count + 1 WHERE id = ?. Average = sum/count, computed at read time (O(1))
>
> [!TIP]
> **🟢 Staff+** — concurrent reviews on a popular business can create a write hotspot on the businesses row. Mitigation: use PG row-level locking for the increment (short lock duration), or batch review aggregation (flush accumulated ratings every 30s from Redis). In ES: rating field updated via the same CDC pipeline — search results always reflect recent ratings
_Why the deep dives connect to the scaling problem: "Multi-dimensional search plus precomputed read data." Each deep dive solves one dimension of the read scaling problem._
Interview script
1. Compound-query script.
2. "Clarifying questions: are we building just search, or also reviews, photos, and business owner tools? And what's the primary query pattern — geo + text combined, or mostly one or the other?"
3. "Good — full local business search with reviews. Primary pattern: geo + text combined — 'best sushi near me.' Core features: search with location, business profiles, reviews. Out of scope: reservations, ads."
4. "The key design decision: Elasticsearch as the single query surface. A 'best sushi near me' query combines full-text matching, geo_distance filtering, rating sorting, and category filtering — all in one request. ES handles all four dimensions natively in a compound query."
5. "Data architecture: PostgreSQL is source of truth for businesses and reviews. ES is a derived read index. Sync via CDC: Debezium reads PG WAL → Kafka → ES indexer. Typical lag: 1-5 seconds. Acceptable for a local search product."
6. "Rating aggregation: maintain a running average in PG — sum_of_ratings and review_count columns. On new review: increment both atomically. Average = sum/count, O(1). Recomputing avg from all reviews at 11K QPS is impossible."
7. "Open now filter: hours data changes every minute by definition. Too dynamic to index. Compute in app layer with 15-min TTL cache per business. Apply as post-ES filter — never in the ES query itself."
8. "Photos: S3 + CDN. On-demand resizing at CDN edge (Lambda@Edge or Imgix). Pre-generate thumbnails at upload time for the most common sizes."
Whiteboard
```
+-------------------+
| Web / Mobile |
| Client |
+---------+---------+
|
v
+-------------------+
| API Gateway |
+----+---------+----+
| |
GET search / view | POST review
| |
v v
+----------------+ +----------------+
| Business | | Review |
| Service | | Service |
+---+--------+---+ +---+--------+---+
| | | |
| | | |
| | | +------------------+
| | | |
| | v v
| | +------------------+ +------------------+
| | | Reviews Table | | Businesses Table |
| | | unique(userId, | | avg_rating |
| | | businessId) | | num_reviews |
| | +---------+--------+ +---------+--------+
| | | ^
| +-------------+--------------------------|
| sync rating update |
| optimistic locking on write |
| |
v |
+-------------------------+ |
| Read Replica / Cache |----------------------------------+
| for hot business reads |
+-----------+-------------+
|
v
+-------------------------+
| Search Store |
| Elasticsearch or |
| Postgres + PostGIS |
| + full text indexes |
+-----------+-------------+
^
|
CDC / async indexing
|
+-----------+-------------+
| Primary DB |
| businesses + reviews |
+-------------------------+
Optional for named locations
+-------------------------+
| Locations Table |
| name -> polygon |
| city / neighborhood |
+-------------------------+
```
The main story you should tell is simple. Search and business reads go through the Business Service, reviews go through the Review Service, the primary database is the source of truth, and search is powered either by Elasticsearch with CDC sync or by Postgres extensions if you want the simpler version. For interviews, I would present the simple version first, then add the search store and location polygons only if the interviewer pushes on search quality or scale.
Upload raw GPS → S3. Async processing worker computes stats and matches segments using Hausdorff distance. TimescaleDB stores GPS time-series. Redis sorted set per segment for leaderboards — ZADD on completion, ZREVRANGE for top rankings.
Problem
Activity tracking, segment performance, and social fitness. Users upload GPS-tracked workouts, get stats, compete on segment leaderboards.
Hard parts: efficient GPS time-series storage, segment matching, and fast leaderboard reads.
Failures
**GPS track upload fails mid-upload (poor cellular connection on a run)**
Partial activity data. User loses their run.
_Fix:_ Client buffers GPS track locally. Chunked upload with resume support (TUS protocol). Client marks upload complete only after server confirms all chunks received and processing started.
**Segment matching job times out for a very long ride (200km, 50K GPS points)**
Leaderboard not updated. User sees no segment efforts after a long ride.
_Fix:_ Async processing with dedicated job queue. Timeout per segment match attempt, not per activity. Break large activities into bounding box segments, parallelize matching. Retry partial failures.
**Redis segment leaderboard grows unbounded**
ZADD/ZREVRANGE on a segment with 10M athletes is slow and memory-heavy.
_Fix:_ Leaderboard is per-segment sorted set. Store only top 10K entries per segment (ZREMRANGEBYRANK after every ZADD to trim). Historical ranking available from PG for display purposes.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 10M athletes, avg 5 activities/week, avg 1 hour at 1 GPS point/sec = 3,600 points/activity |
| Read QPS | 10M × 5 / 7 / 86400 ≈ 83 activity uploads/s but: 83 × 3,600 GPS points = 300K GPS point writes/s to TimescaleDB |
| Write QPS | 300K GPS points/s is the dominant write load — TimescaleDB's append-only model handles this well |
| Storage | 10M users × 5 activities/week × 52 weeks × 3,600 points × 24 bytes (lat,lon,time,elevation) ≈ 45 TB GPS data/year |
| Cache math | 10K active segments × top-10K entries × 8 bytes = 800 MB leaderboard data in Redis — manageable |
| Verdict | GPS write volume (300K/s) is larger than it looks. TimescaleDB's time-partitioned append model is essential. Random-access DB would struggle. |
Design decisions
**TimescaleDB vs. Cassandra vs. InfluxDB for GPS time-series**
→ TimescaleDB (PostgreSQL extension)
GPS data is time-series but also needs joins with activity metadata, segment boundaries, and athlete data. TimescaleDB gives time-series performance with full SQL. Cassandra would require denormalization of all joins.
_Revisit when:_ InfluxDB if write volume exceeds TimescaleDB limits (~500K points/s per node).
**Sync vs. async segment matching**
→ Async via Kafka processing job
Segment matching is CPU-intensive (Hausdorff distance over 50K points against thousands of segments). Doing it synchronously would make activity uploads take minutes. User gets immediate upload confirmation, segment efforts appear within minutes.
_Revisit when:_ Real-time matching possible for popular segments only (cache segment geometry, match inline).
**Polyline encoding client-side vs. server-side**
→ Client-side encoding before upload
Reduces upload payload by 75%. Client CPU cost is negligible (mobile has fast FPUs). Encoded polyline is still decodable server-side for processing.
_Revisit when:_ Server-side encoding if client implementation is buggy across platforms (just send raw and compress server-side).
Follow-up Q&A
**How do you handle GPS noise (jumpy coordinates during tunnel)?**
Kalman filter or simple smoothing on the GPS stream client-side before upload. Server-side validation: reject points with speed > 200km/h (teleporting GPS artifact). Flag activities with high noise ratio for manual review.
**How would you add live activity tracking (share live location with friends)?**
Separate real-time pipeline: client streams GPS every 5s to a WebSocket server. Friends who are watching subscribe via SSE. Store live track in Redis (not TimescaleDB — too much write pressure). Persist to TimescaleDB async. This is the Strava Beacon feature.
**How do you prevent segment leaderboard manipulation?**
Statistical outlier detection: flag segment efforts with impossible speed (segment KOM for a 20km segment in 5 minutes = 240 km/h on a bike — suspicious). Cross-reference with heart rate and power data if available. Manual review queue for suspected anomalies.
**How do you handle athletes deleting activities?**
Soft delete the activity. Asynchronously remove all segment efforts for that activity from Redis leaderboards (ZREM). Remove GPS points from TimescaleDB (expensive — runs as background job). Leaderboard is eventually consistent after deletion.
**What's your data retention policy for GPS tracks?**
Hot storage (TimescaleDB): full resolution GPS for 2 years. Cold storage (S3): compressed polyline for lifetime. Old GPS data archived to S3 Glacier after 2 years. User-facing: always show full track (decompress on demand from S3 for old activities).
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: throughput, queue lag, cache effectiveness. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: async replication lag <30s; failover promotes read replica. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — PG for all activity data. GPS as JSON blob in one column. Basic stats computed on upload. No segment matching. Works for 100K athletes.
**v2 — Time-series + segments** — TimescaleDB for GPS points. Async Kafka segment matching. Redis leaderboards. Polyline encoding. Live activity sharing. Handles 10M athletes.
**v3 — Social + ML** — Fitness trend analysis. Training load ML model. Route recommendations. Group challenges. Beacon (live tracking). Global athlete clustering for social features.
Why it's hard to scale
The hard part in Strava is not raw request volume. It is that the product mixes offline tracking, large route data, and social reads in one system.
There are three pain points you should call out. First, each activity generates a long stream of GPS points, so storage grows fast and older route data can get expensive. Second, the app must work with weak or no connectivity, which means the client has to buffer data locally and sync later without losing too much progress. Third, if you add live sharing, the system becomes much harder because now you are handling frequent location updates plus many friends reading those updates at the same time.
A good interview summary is this. Strava is hard because the client does a lot of the work, route data is much heavier than normal app metadata, and real time sharing can turn a simple upload system into a continuous update system.
Key points
- **Scope it first** — Core: upload GPS activity, compute stats, match segments, leaderboards, social feed. Out of scope unless asked: live tracking, route planning, training analytics, coaching.
- **Upload is async — always** — Client sends polyline-encoded GPS track to S3. Upload API acknowledges immediately. Kafka job triggers async processing. Stats and segment matches appear minutes later — that is the expected UX.
- **TimescaleDB for GPS time-series** — 300K GPS points/sec appended across all users. TimescaleDB: time-partitioned, columnar compression (5× smaller), SQL joins with activity metadata. Never use plain PostgreSQL for this volume.
- **Segment matching — Hausdorff distance** — Spatial index on segment bounding boxes (R-tree or geohash). For each activity: query segments whose bounding box overlaps. Then run Hausdorff distance check on candidates only. Reduces 1M segments to ~200 candidates per activity.
- **Leaderboard — Redis sorted set with trimming** — ZADD leaderboard:{segment_id} elapsed_seconds user_id. ZREMRANGEBYRANK after each ZADD to cap at top 10K entries. ZREVRANGE for display. Historical beyond top 10K fetched from PostgreSQL.
- **GPS noise handling** — Douglas-Peucker simplification or Kalman filter client-side before upload. Server-side: reject points implying speed > 200 km/h. Noisy tracks create false segment efforts — flag for review.
- **Failure mode to name** — Segment matching job fails mid-activity: idempotent Kafka consumer replays. Track transcode status per (activity_id, segment_id). Retry only failed segments, not the whole activity.
> Mental model: async pipeline. Upload to S3, process via Kafka worker, write to TimescaleDB + Redis leaderboards.
Tradeoffs
**Sync vs async processing** — Async decouples upload from computation — upload confirmation is instant. Tradeoff: leaderboard and stats appear minutes later, not immediately on save.
**Redis sorted set vs DB query for leaderboard** — Redis ZREVRANGE is O(log N + K) in memory — sub-millisecond. A DB query scanning all efforts for a segment at 83 uploads/sec would be too slow.
**TimescaleDB vs Cassandra for GPS time-series** — TimescaleDB gives full SQL joins with activity metadata — needed for segment matching queries. Cassandra requires denormalizing all joins upfront. TimescaleDB is the right call when structured queries matter.
**Bloom filter vs exact set for segment dedup** — A user accumulates ~73K swipes/2yr. Bloom filter at 1% false positive is O(1), 730KB RAM — fast and cheap for deck generation. Exact Cassandra check is only needed for match creation where a false negative is unacceptable.
> "Upload decoupled from processing via S3 + Kafka. Polyline encoding cuts storage. Redis sorted sets for leaderboards."
Deep dives
#### Deep dive 1: GPS time-series storage — TimescaleDB design for append-only high-volume data
> [!CAUTION]
> **🔴 Weak** — store GPS points as rows in a PostgreSQL table indexed by activity_id
>
> [!WARNING]
> **🟡 Strong** — the core constraint is 300K GPS point writes/second (10M users × 5 activities/week × 3,600 points/activity) combined with complex time-range queries for activity display and segment analysis. Weak answer: PostgreSQL with a GPS_points table. Strong answer: TimescaleDB with a hypertable partitioned by (activity_id, time). Time-partitioned storage means: (1) recent data is in memory or SSD, old data on cheaper storage; (2) time-range queries touch only the relevant partitions; (3) TTL/retention policies drop old partitions cheaply (DROP is instant, DELETE is O(N))
>
> [!TIP]
> **🟢 Staff+** — design: GPS_points table has (activity_id, timestamp, lat, lng, elevation, heart_rate, power) as a time-series table. The partition key must be on timestamp for range query efficiency. Compression: TimescaleDB columnar compression reduces GPS data to 20% of original size with no query changes. Rollup: pre-aggregate GPS points to 10-second intervals for activities older than 30 days — reduces storage by 10× at the cost of reduced resolution for old activities (acceptable — users rarely scroll old activity GPS at full resolution)
#### Deep dive 2: Segment matching — Hausdorff distance at scale
> [!CAUTION]
> **🔴 Weak** — iterate through all 1M segments and run Hausdorff distance on each for every uploaded activity
>
> [!WARNING]
> **🟡 Strong** — every activity must be matched against all segments whose bounding box overlaps the activity's bounding box. At 83 activity uploads/second and 1M segments, naive O(activities × segments) is impossible
>
> [!TIP]
> **🟢 Staff+** — approach: spatial index on segment bounding boxes (R-tree or geohash grid). For each activity: (1) compute activity bounding box, (2) query spatial index for segments whose bounding box overlaps, (3) run Hausdorff distance check on the reduced candidate set. The spatial index reduces candidates from 1M to ~100-500 per activity. Hausdorff distance: the maximum of the minimum distances between two polylines. Two polylines match if Hausdorff distance < threshold (e.g., 25 meters). GPS noise handling: smooth the activity GPS track with a Kalman filter or Douglas-Peucker simplification before matching — reduces false negatives from noisy data. Parallel matching: each activity's segment matches can be computed independently — embarrassingly parallel across worker nodes, partitioned by activity bounding box geohash
#### Deep dive 3: Segment leaderboards — Redis sorted sets with trimming
_Each segment has a leaderboard: fastest time by athletes who have completed that segment. The access pattern: frequent reads (users checking their ranking), moderate writes (new segment efforts after activity processing)_
> [!CAUTION]
> **🔴 Weak** — Oversimplify segment leaderboards — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Each segment has a leaderboard: fastest time by athletes who have completed that segment. The access pattern: frequent reads (users checking their ranking), moderate writes (new segment efforts after activity processing)
>
> [!TIP]
> **🟢 Staff+** — design: Redis sorted set per segment_id (key: leaderboard:{segment_id}, score: elapsed_seconds, member: user_id). ZADD adds a new effort; if the user has a previous effort, ZADD with NX flag only adds new members, so first add only — use regular ZADD but also track the user's best time separately to avoid storing worse efforts. On ZADD, also check if user's new time is better than their stored best (ZSCORE lookup, O(log N)), and only update if improved. ZREVRANGE for top-N is O(log N + K). Leaderboard trimming: ZREMRANGEBYRANK removes bottom entries, keeping only top 10K per segment (50K × 10K × 8 bytes = 4 GB total Redis for all segments). Historical rankings beyond top 10K are fetched from PostgreSQL
_Why the deep dives connect to the scaling problem: "Offline tracking, large route data, and social reads." Deep dive 1 solves time-series storage. Deep dive 2 solves segment matching at scale. Deep dive 3 solves leaderboard performance._
Interview script
1. Async-pipeline script.
2. "Clarifying questions: are we designing the full platform — upload, segment matching, social feed, live tracking — or just the core activity pipeline?"
3. "Good — upload plus segment matching plus leaderboards. Core features: upload GPS activity, compute stats, match segments, update leaderboards. Out of scope: live tracking, coaching, route planning."
4. "Upload is always async: client sends polyline-encoded GPS track to S3. API acknowledges immediately. Kafka event triggers async processing. Users expect stats and segment matches to appear within minutes — not blocking the upload."
5. "GPS storage: TimescaleDB. 300K GPS points/sec is an append-only time-series workload. TimescaleDB: time-partitioned, columnar compression (5× smaller), full SQL for joins with activity metadata. Plain PostgreSQL cannot handle this write volume."
6. "Segment matching: spatial index (R-tree or geohash) on segment bounding boxes. For each activity, query segments whose bounding box overlaps. Then run Hausdorff distance on the ~200 candidate segments only. Without the spatial index, checking all 1M segments per activity is impossible."
7. "Leaderboards: Redis sorted set per segment_id. ZADD new effort, ZREMRANGEBYRANK to cap at top 10K. ZREVRANGE for display. Personal ranking uses ZRANK. Historical beyond top 10K is fetched from PostgreSQL on demand."
8. "Key tradeoff: async processing means the leaderboard is not updated instantly on upload. A user who sets a segment PR sees it reflected a few minutes later. This is the acceptable tradeoff for decoupling upload latency from segment matching computation."
Whiteboard
```
+----------------------+
| Mobile App |
| start pause stop |
| local GPS tracking |
| local stats display |
| offline buffer |
+----------+-----------+
|
create sync fetch activities
|
v
+----------------------+
| API or LB Layer |
+----------+-----------+
|
v
+----------------------+
| Activity Service |
| activity lifecycle |
| ingest route uploads |
| fetch activity feed |
+-----+-----------+----+
| |
| |
v v
+------------------+ +------------------+
| Activities DB | | Friends DB |
| activity metadata | | user friendships |
| route points | +------------------+
| state log |
+------------------+
|
|
v
+------------------------------+
| optional cache for hot reads |
+------------------------------+
Read flow
Mobile App -> API -> Activity Service -> Activities DB or Friends DB
Write flow for normal offline-first design
Mobile App stores GPS locally during run
Mobile App uploads completed activity in one sync
Activity Service writes metadata and route data to DB
Optional realtime sharing extension
athlete app sends periodic updates every few seconds
|
v
+----------------------+
| Activity Service |
+----------+-----------+
|
v
+----------------------+
| Activities DB |
+----------+-----------+
|
friends poll for latest activity updates
|
v
+----------------------+
| Friends Apps |
+----------------------+
```
The main idea is that the client does most of the live tracking work locally. That is the key simplification here. The backend mainly handles activity creation, final sync, and reads for completed runs. If you want, I can also give you a cleaner interview-style version that fits in 30 seconds on a whiteboard.
Online auction (eBay) Medium
Contention · Bid ordering · Real-time updates
Redis Lua CASPostgreSQLKafkaWebSocket
Bids serialized via Redis Lua compare-and-set: atomically check if new bid exceeds current max, update if yes. Validated bid written to PostgreSQL for durability. Kafka fans out to WebSocket subscribers showing the live bid feed.
Problem
Accepting bids correctly at scale. Ensuring only valid higher bids win, no bids are lost, and everyone sees the current highest bid quickly.
Failures
**Redis goes down mid-auction**
No authoritative current bid state. New bids can't be validated. Auction is paused.
_Fix:_ Redis Sentinel for HA. On Redis failure: freeze auction (stop accepting new bids), reconstruct highest bid from PostgreSQL (source of truth), resume when Redis recovers. SLA: < 30s pause.
**Sniping (bid placed in last 3 seconds)**
Most bidders lose because they can't react in time. Bad user experience, low trust.
_Fix:_ Auction extension: if a bid is placed in last 3 minutes, extend auction by 3 minutes (eBay's actual behavior). Prevents pure sniping while still allowing competitive last-minute bidding.
**Bid fan-out overwhelms WebSocket servers during auction close**
Bid in last 30 seconds triggers thousands of simultaneous WebSocket pushes. Server falls over.
_Fix:_ Batch WebSocket pushes with 100ms coalescing window (same as FB Live Comments). Individual bid events merged into a single update payload. At extreme scale, SSE over WebSocket (simpler fan-out).
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 10M active auctions, 100K active bidders at peak, 1 bid/10s per active bidder |
| Read QPS | Live updates: 100K watchers × poll/5s = 20K fan-out messages/s peak during popular auction close |
| Write QPS | 100K bidders / 10s = 10K bid write QPS — each is a Redis Lua CAS + PG write |
| Storage | Bid history: 10M auctions × avg 50 bids × 100 bytes ≈ 50 GB PG — small |
| Cache math | Current max bid per auction: 10M × 50 bytes ≈ 500 MB Redis — fits easily on single Redis instance |
| Verdict | Fan-out of bid updates to watchers is the scaling challenge, not bid storage. Coalescing updates is essential for popular auctions. |
Design decisions
**Redis Lua CAS vs. database transactions for bid atomicity**
→ Redis Lua CAS as fast path, PG as durable record
PG row lock for bid validation adds 5-20ms. Redis Lua CAS adds <1ms. Under high contention (auction closing), Redis wins on latency. PG is written after Redis CAS succeeds — durability is not sacrificed, just sequenced.
_Revisit when:_ PG advisory lock as fallback when Redis is unavailable.
**Proxy bid (auto-bid) design**
→ Proxy bid stored server-side, current bid incremented automatically
User sets a max price. System auto-bids on their behalf, incrementing by minimum bid increment, up to their max. This is eBay's proxy bidding. It's a state machine that runs server-side, not client-side.
_Revisit when:_ This requires careful treatment — proxy bid amounts are private data. Stored encrypted in PG, never exposed via API.
**Auction close time precision**
→ Scheduled job with 1-second precision, not millisecond
Auction close is user-facing time (12:00 PM). 1-second granularity is what users expect. Sub-second precision creates unfair advantages for clients with low-latency connections.
_Revisit when:_ Exact-time close for high-frequency trading-style auctions (different use case, different design).
Follow-up Q&A
**Two bids arrive at exactly the same time — which wins?**
Redis is single-threaded. Commands execute serially. The bid whose EVAL command reaches Redis first wins. This is deterministic, fair (first-come), and requires no additional coordination.
**How do you handle a bidder who wins but doesn't pay?**
Reserve winning bid, send payment invoice. Payment window (24-48hr). If unpaid: cancel sale, offer to next-highest bidder (second-chance offer). Track non-payment rate per bidder — suspend bidding privileges after threshold.
**How do you prevent bid retraction abuse (bid high, retract, bid lower)?**
Allow retraction only under specific conditions (item description materially wrong). Track retraction count per bidder. High retraction rate → bidder restriction. Log all retraction events for fraud analysis.
**How would you scale to 1B concurrent auctions?**
Auctions are independent — shard by auction_id. Each shard handles its own Redis CAS and PG writes. No cross-auction coordination needed. Fan-out delivery servers also shard by auction_id range. Near-linear horizontal scale.
**How do you handle currency and international bidding?**
Store all bids in a base currency (USD). Display in local currency at read time using exchange rate service (cached, 1-min TTL). Minimum bid increment calculated in base currency to avoid floating-point issues across currencies.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: throughput, queue lag, cache effectiveness. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: async replication lag <30s; failover promotes read replica. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — PG for everything. SELECT FOR UPDATE for bid lock. Simple polling for live updates. Works for low-volume auctions.
**v2 — Real-time + correctness** — Redis Lua CAS for fast bid validation. WebSocket for live updates. Kafka bid events. Proxy bidding. Auction extension for sniping. Handles peak auctions.
**v3 — Global scale** — Sharded by auction_id. Regional deployment. Fraud detection ML model. Anti-shill bidding detection. Mobile-optimized push notifications.
Why it's hard to scale
The hard part is contention on a tiny piece of shared state. Many users may bid on the same auction at nearly the same time, but only one current highest bid can be correct.
That creates three scaling pain points. First, bid writes are hot and correctness matters, so you need atomic updates or version checks to avoid accepting stale lower bids. Second, auctions get bursty near the end, so one popular item can suddenly get hammered even if overall traffic looks manageable. Third, users expect live updates, which means one accepted bid may need to fan out quickly to many watchers across many servers. So the short interview answer is that Online Auction is hard because it combines hotspot writes, strict bid consistency, and real time fan-out.
Key points
- **Scope it first** — Core: list item, place bid, real-time bid feed, auction close, winner determination. Out of scope unless asked: payments, shipping, dispute resolution, seller ratings.
- **Atomic bid check — Redis Lua CAS** — GET current_max, compare, SET if higher — all in one Lua script. Single-threaded Redis execution means no two bids can race. Never use GET + SET as separate operations.
- **Redis for speed, PostgreSQL for truth** — Redis holds current max bid for sub-millisecond reads during live auction. PostgreSQL stores every bid for audit, dispute resolution, and winner determination. Both required.
- **Auction close is a state machine** — ACTIVE → CLOSING (stop accepting bids) → CLOSED (determine winner). Close job must be idempotent: UPDATE SET status=CLOSED WHERE status=CLOSING. If it runs twice, second is a no-op.
- **Anti-sniping extension** — Bid placed in last 3 minutes → extend auction by 3 minutes. UPDATE auctions SET ends_at = ends_at + INTERVAL 3 minutes WHERE id=? AND ends_at - NOW() < 3 minutes. Users prefer fair competition over fixed end time.
- **Real-time bid feed — SSE with coalescing** — 1K watchers × bid storm in last 30s = massive fan-out. 100ms coalescing: push current max bid state, not every individual bid. Watchers see the current price, not a log of every increment.
- **Failure mode to name** — Redis goes down mid-auction: freeze auction (reject new bids), reconstruct highest bid from PostgreSQL bid history, resume when Redis recovers. SLA: < 30s pause. PostgreSQL is always authoritative.
> Mental model: Redis for speed and ordering, PostgreSQL for durability, Kafka for live feed.
Tradeoffs
**Redis CAS vs DB transaction for bid lock** — Redis Lua CAS is sub-millisecond under high bid contention. DB SELECT FOR UPDATE adds 5–20ms and serializes under load. Redis wins on latency; PG write must follow for durability.
**WebSocket vs polling for live bid feed** — WebSocket gives real-time updates with no wasted requests. SSE is simpler for one-directional push and auto-reconnects. Both beat polling. SSE is the right default for a bid feed.
**Hard close vs auction extension anti-sniping** — Hard close at the scheduled time is simple but rewards sniping tools. Extension (add 3 min on any bid in last 3 min) levels the field and increases final price — tradeoff is unpredictable end time, which some bidders dislike.
**Proxy bidding server-side vs client-side** — Server-side proxy bidding (auto-bid up to user max) prevents the user from revealing their max price to competitors. Client-side is simpler but the user must stay online. Server-side is the correct model for a real auction.
> "Redis for speed and ordering, PostgreSQL for durability. Never rely on Redis alone for financial data."
Deep dives
#### Deep dive 1: Concurrent bid handling — atomic compare-and-set for bid correctness
_Two bidders submit simultaneously: A bids $100, B bids $102, both at the exact same millisecond. The system must accept $102 and reject $100_
> [!CAUTION]
> **🔴 Weak** — database transaction with SELECT FOR UPDATE
>
> [!WARNING]
> **🟡 Strong** — Redis Lua CAS (Compare-And-Set): (1) GET current_highest_bid, (2) if new_bid > current_highest, SET new_bid, return accepted, else return rejected. All atomic in one Lua script
>
> [!TIP]
> **🟢 Staff+** — Database transaction with SELECT FOR UPDATE is also correct but adds 5-20ms PG lock latency under high contention. Redis Lua CAS is <1ms. The tradeoff: Redis is not durable by default (RDB snapshot may be seconds old), so every accepted bid must also be written to PostgreSQL before returning success. The two-write pattern: Redis for speed and ordering, PG for durability. If Redis and PG diverge (Redis accepts a bid but PG write fails): on startup/recovery, the PG bid history is authoritative. Redis state is reconstructed from PG
#### Deep dive 2: Auction close — preventing last-millisecond races
> [!CAUTION]
> **🔴 Weak** — close the auction at exactly the scheduled time and reject bids that arrive after
>
> [!WARNING]
> **🟡 Strong** — auction close has a subtle correctness problem: a bid submitted at 23:59:59.999 and processed at 00:00:00.001 — is it valid? Weak answer: ignore bids after close time. Strong answer: explicit auction state machine: ACTIVE → CLOSING → CLOSED. On scheduled close time: set state to CLOSING (still accepts bids for a brief grace period). After grace period: set to CLOSED, reject all new bids. Determine winner from highest bid in PG
>
> [!TIP]
> **🟢 Staff+** — the auction close job must be idempotent — if it runs twice (due to retry), it should produce the same result. Implement with optimistic concurrency: UPDATE auctions SET status='CLOSED', winner_id=? WHERE id=? AND status='CLOSING'. If zero rows updated: another instance already closed it. Auction extension (eBay's approach): if a bid arrives in the last 3 minutes, extend the auction by 3 minutes. Implement: on any bid in the last 3 minutes, UPDATE auctions SET ends_at = ends_at + INTERVAL '3 minutes' WHERE id=? AND ends_at - NOW() < INTERVAL '3 minutes'
#### Deep dive 3: Real-time bid feed — fan-out to watchers without overloading
> [!CAUTION]
> **🔴 Weak** — push every individual bid event to all watchers in real-time
>
> [!WARNING]
> **🟡 Strong** — an active auction with 10,000 watchers × 1 new bid every 10 seconds = 1,000 SSE messages/s for one auction. Naive fan-out: one Kafka message → 10,000 individual SSE pushes
>
> [!TIP]
> **🟢 Staff+** — delivery servers partitioned by auction_id (consistent hash). Each delivery server holds all SSE connections for its auction range. On new bid event: Kafka consumer on that delivery server pushes to all 10,000 connected clients in a tight loop. No cross-server coordination needed. For the last 30 seconds of a popular auction (bid storm): coalesce — buffer bids in 100ms windows, push the latest state (current highest bid) rather than every individual bid. Clients don't need every intermediate bid — they need the current highest bid to display correctly. This reduces fan-out from O(bids_per_second × watchers) to O(10 × watchers) during the final sprint
_Why the deep dives connect to the scaling problem: "Hotspot writes, bid consistency, and real-time fan-out." Each deep dive addresses one dimension._
Interview script
1. Atomic-bid script.
2. "Clarifying questions: are we building a general auction platform, or focused on a specific model — English auction (ascending price), Dutch (descending), sealed bid? And what's the real-time requirement for the bid feed?"
3. "Good — English auction, real-time bid feed. Core features: list item, place bid, real-time bid updates, auction close and winner determination. Out of scope: payments, shipping, dispute resolution."
4. "The hard problem: two bidders submit simultaneously. The system must accept exactly the higher bid with no race condition. This drives the core design."
5. "Bid acceptance: Redis Lua CAS. GET current_max_bid, compare, SET if higher — all in one atomic Lua script. Single-threaded Redis execution: no race possible. Every accepted bid is also written to PostgreSQL for durability and audit."
6. "Auction close state machine: ACTIVE → CLOSING → CLOSED. The close job is idempotent: UPDATE SET status=CLOSED WHERE status=CLOSING. If it runs twice, the second is a no-op. Winner = highest bid in PostgreSQL at close time."
7. "Anti-sniping: bid in the last 3 minutes → extend auction by 3 minutes. One SQL update. Users prefer a fair final sprint over a fixed end time that rewards sniping tools."
8. "Real-time bid feed: SSE with 100ms coalescing. During a bid storm in the last 30 seconds, push the current max bid state, not every individual increment. Watchers need the current price, not a log of every $1 raise."
Whiteboard
```
+-------------------+
| Clients |
| web mobile browser |
+---------+---------+
|
GET POST /auctions
POST /bids
SSE bid updates
|
+---------v---------+
| API Gateway |
| auth routing rate |
+----+----------+----+
| |
read path | | write path
| |
+--------------v--+ +--v----------------+
| Auction Service | | Bid Ingest API |
| auction details | | accept bid fast |
+-------+---------+ +---------+---------+
| |
| |
+--------v---------+ |
| Auctions DB |<---------------+
| auctions items | optional direct read
| max_bid on row |
+--------+---------+ |
^ |
| v
| +--------+---------+
| | Kafka |
| | topic partitioned|
| | by auctionId |
| +--------+---------+
| |
| v
| +--------+---------+
| | Bidding Service |
| | consumer workers |
| | validate bid |
| | OCC or row lock |
| +---+----------+---+
| | |
| | |
| write bid history |
| | |
| v v
| +------+--+ +---+----------------+
| | Bids DB | | Pub Sub / Fanout |
| | history | | broadcast updates |
| +---------+ +---+----------------+
| |
| |
+------v----------------------------------v------+
| SSE / Realtime Gateway instances |
| keep client connections by auctionId |
| push latest accepted max bid to watchers |
+----------------------+-------------------------+
|
v
+----+----+
| Clients |
| live UI |
+---------+
```
The key idea is this. Reads and writes are split. Auction Service handles viewing and creating auctions. Bids go through a durable queue first so you do not lose them under spikes. Then Bidding Service processes bids in order per auction, updates the auction row max bid safely, stores bid history, and publishes the new highest bid to the realtime layer.
If you need to simplify this in an interview, keep the core path to four boxes. Client to API Gateway to Kafka to Bidding Service to Database, plus SSE for live updates. That shows you understand the two hard parts, which are correct bid acceptance and real-time fanout.
A distributed crawler fleet scrapes prices. A consistent hash scheduler assigns products to crawlers stably. Price changes publish to Kafka then fan out to: the TSDB for storage and the Alert Service for watchlist evaluation.
Problem
Helping users know whether an Amazon item is a good deal. Collect prices over time, show history charts, and alert when price drops below a threshold.
The hard part: getting accurate price data for a huge product catalog at scale.
Failures
**Amazon blocks the crawler (IP ban)**
Price data for all products stops updating. Users miss deals. Trust in the service erodes.
_Fix:_ IP rotation pool (residential proxies). Respect crawl delays. Randomize request timing and user agent strings. Chrome extension as distributed crowdsourced price collection bypasses this entirely for high-priority products.
**Alert fanout overwhelms email service during a major sale (Amazon Prime Day)**
1M users have alerts set for discounted products. All fire simultaneously. Email service throttled.
_Fix:_ Alert queue with rate limiting (SQS). Partition by alert priority (price drop % determines urgency). Batch similar alerts per user (one email per user with all triggered alerts vs. N separate emails).
**Price history TimescaleDB query slow for old products**
3-year price chart takes 5 seconds to load for a product tracked since launch.
_Fix:_ Pre-aggregate price history: daily min/max/avg stored alongside raw data. Charts use aggregated data for time ranges > 30 days. Raw data only for recent window. Query time drops from seconds to milliseconds.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 500M products tracked, 1% re-crawled daily = 5M crawls/day, 10M users, 20% have active alerts |
| Read QPS | 10M users × 5 price chart views/day / 86400 ≈ 579 chart read QPS — served from TSDB + cache |
| Write QPS | 5M crawls / 86400 ≈ 58 price updates/s — tiny, but each write triggers alert evaluation |
| Storage | 500M products × 365 days × 2 prices (min/max) × 8 bytes ≈ 2.9 TB/year TSDB — manageable |
| Cache math | Top 1M products get 90% of chart views. Cache price history snapshots: 1M × 10KB = 10 GB Redis. Serves 90% of chart reads from cache. |
| Verdict | Alert fanout on sale events is the peak load problem. 2M alerts firing simultaneously needs queue + rate limiting. Normal operation is low QPS. |
Design decisions
**Extension-first vs. crawler-first data collection**
→ Chrome extension as primary for user-active products, crawler as secondary
Extension: zero crawl cost, real-time prices when users browse Amazon, no rate limiting concerns. Covers top 1M products users actually care about. Crawler: covers the long tail but at much lower frequency.
_Revisit when:_ Extension alone has coverage gaps (products no active user browses). Crawler is essential for the full catalog.
**Alert evaluation: per-write vs. scheduled polling**
→ Event-driven: Kafka price-change event triggers alert evaluation
Polling all 100M active alerts every 5 minutes = 333K DB queries/s. Event-driven: only evaluate alerts for products that actually changed price (58/s × avg 5 alerts/product = 290 evaluations/s). 1000× more efficient.
_Revisit when:_ Scheduled polling for users who set time-based alerts (e.g., 'alert me if price drops by Monday').
**TSDB vs. PG time-series for price history**
→ InfluxDB / TimescaleDB
Price history is pure time-series: one price per product per time. Never updated (immutable). Queried by product + time range. TSDB's columnar time-partitioned storage gives 10× better query performance than PG for this access pattern.
_Revisit when:_ PG with proper indexing works up to ~50M products. Switch to TSDB when query latency becomes noticeable.
Follow-up Q&A
**How do you detect a fake price drop (item marked up then 'discounted')?**
Price history is the answer — you can see the baseline. ML model trained on price history patterns: flag items where 'original price' was only briefly at that level. Show price history chart prominently. Let users judge.
**How would you add support for multiple retailers (BestBuy, Walmart)?**
Abstract crawler behind a retailer interface. Each retailer has its own rate limits, HTML parser, and price extraction logic. Price is always normalized to a canonical schema (product_id, retailer_id, price, currency, timestamp). Product matching across retailers is a separate hard problem (same product, different SKUs).
**How do you keep price data fresh for the most popular products?**
Freshness priority queue: products with the most active alerts and highest user interest get crawled most frequently (every 5 min). Long tail products crawled daily. Extension users provide real-time updates for products they browse, bypassing the crawl priority queue entirely.
**What's your data model for products across multiple retailers?**
Products table: (product_id, canonical_name, category, created_at). Listings table: (listing_id, product_id, retailer_id, retailer_sku, url). Prices table: (listing_id, price, currency, timestamp). One product maps to many listings across retailers. Alerts are on listing_id, not product_id.
**How do you handle price errors (data scraping bugs returning wrong prices)?**
Price validation: flag if new price is >50% different from previous price (likely scraping error). Hold for manual review or second-source confirmation before storing. Never trigger alerts on potentially bad data. Log all validation failures for crawler improvement.
**How do you avoid re-alerting on the same price drop?**
Track last_alert_sent_at and last_alert_price per watchlist. Re-alert only when price recovers above threshold then drops again, or drops by another meaningful delta (e.g., 5%). Prevents notification spam on hover pricing.
**How do you prioritize crawl budget across 500M products?**
Priority queue scored by active alerts + extension page views + sales rank. Top 1M products crawled every 5–15 min; long tail daily or weekly. Budget is a product decision expressed as crawl slots per hour.
**How do you serve price history charts fast for 3-year ranges?**
TimescaleDB continuous aggregates: raw for 7 days, daily min/max for 90 days, weekly for multi-year. Chart API selects rollup tier by requested range — 170× fewer points with no visible quality loss.
Evolution
**v1 — MVP** — Simple crawler polls Amazon hourly for top 10K products. PG stores price history. Email alerts via cron. Works for early users.
**v2 — Scale** — Chrome extension for crowdsourced prices. TSDB for price history. Kafka event-driven alerts. Consistent hash crawler fleet. Handles 50M products.
**v3 — Intelligence** — ML fake-discount detection. Deal score model. Price prediction. Multi-retailer support. Browser extension for all major browsers. Affiliate revenue tracking.
Why it's hard to scale
The hard part is not storing prices. It is collecting and updating them at huge scale without overwhelming Amazon or your own system.
There are three main pain points. First, data collection is constrained. You may want to track 500 million products, but you cannot crawl them all frequently because Amazon rate limits scraping, so freshness becomes a resource allocation problem. Second, the workload is very uneven. A small set of products matters a lot more than the long tail, so you need prioritization based on user interest or extension traffic instead of treating every product the same. Third, one price change can create downstream fan-out. A single update may trigger validation, storage, chart updates, and notifications to many subscribers, so the system is not just a crawler. It is also an event processing pipeline.
A good interview summary is this. Price Tracking Service is hard to scale because data ingestion is externally constrained, freshness matters, and each accepted price update can fan out into a lot of follow-on work.
Key points
- **Scope it first** — Core: track product prices, store history, trigger alerts when price crosses threshold, display price charts. Out of scope unless asked: affiliate links, deal scoring, ML price prediction.
- **Two collection strategies** — Chrome extension: real-time, zero crawl cost, covers products users actively browse. Crawler: covers the long tail at lower frequency. Extension data for top 1M products, crawler for the rest.
- **TSDB for price history — not SQL** — Every price change is an immutable append. Queried by (product_id, time_range). InfluxDB or TimescaleDB: columnar, time-partitioned, compression. Plain PostgreSQL at 58 writes/sec × 500M products = wrong tool.
- **Multi-resolution rollups** — Raw price events: keep forever (small — only changes stored, not every poll). Daily rollup: min/max/avg per product per day. Chart rendering: raw for 7-day view, daily for 90-day, weekly for multi-year. Pre-computed by TimescaleDB continuous aggregates.
- **Event-driven alerts** — Price change → Kafka event → Alert Service evaluates watchlists for that product_id only. At 58 changes/sec × 5 alerts/product = 290 evaluations/sec. 1000× cheaper than polling all 100M alerts every 5 min.
- **Trust but verify** — Extension data can be wrong: A/B test pricing, regional variants, scraping errors. Fast-accept for display. Flag if new price differs >50% from previous. Priority-crawl flagged products for verification before storing permanently.
- **Failure mode to name** — Crawler gets IP-banned: rotate residential proxies, respect Crawl-delay, randomize timing. Extension-first strategy means the most important products still get real-time data even if crawler is blocked.
> Open with the data collection split: extension gives real-time coverage for user-active products, crawler covers the long tail. This framing immediately shows you understand the scale constraint.
Tradeoffs
**Coverage vs freshness** — Crawling more products = better coverage, but 500M products cannot all be kept fresh. Answer: extension provides real-time data for user-active products; crawler handles the long tail at lower frequency.
**Polling vs event-driven for alerts** — Polling all 100M active alerts every 5 min = 333K DB queries/sec. Event-driven evaluation triggers only on price-change Kafka events — 290 evaluations/sec at 58 changes/sec. 1000× more efficient.
**Raw data vs rollups for price history charts** — Raw data for a 3-year chart = 26K points per query at full resolution. Daily rollups for >30-day ranges reduce this to 1,095 points — 24× faster query with no visible chart difference at typical chart widths.
**Trust extension data vs verify** — Extension data from user browsers is cheap and real-time but can be wrong (A/B test prices, regional variants). Fast-accept for display, but flag >50% change deviations for crawler re-verification before storing permanently.
> Coverage vs freshness, polling vs event-driven, raw data vs rollups. My defaults: extension-first, event-driven alerts, multi-resolution TSDB. Each choice is driven by the read/write asymmetry.
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
#### Deep dive 1: Data collection strategy — extension + crawler with trust-but-verify
> [!CAUTION]
> **🔴 Weak** — crawl Amazon every hour for all tracked products
>
> [!WARNING]
> **🟡 Strong** — two-tier collection — Chrome extension gives real-time prices for user-active products at zero crawl cost; crawler covers the long tail at lower frequency. The extension is the primary path for the top 1M products users actually care about
>
> [!TIP]
> **🟢 Staff+** — extension data can be wrong — A/B test prices, regional variants, scraping bugs. Fast-accept for display, but flag price changes >50% from previous as anomalies and trigger a priority crawler re-fetch before storing permanently. This two-layer validation gives low latency for normal updates and high confidence for outliers
#### Deep dive 2: Event-driven alert evaluation — Kafka over polling
> [!CAUTION]
> **🔴 Weak** — cron job every 5 minutes queries all 100M active alerts for products that changed price. This is O(active_alerts) per run = 333K DB queries per second just for alerts. Unacceptable
>
> [!WARNING]
> **🟡 Strong** — event-driven evaluation. When a price update is stored, publish a Kafka event {product_id, old_price, new_price}. Alert service consumes the event and evaluates only alerts for that product_id. At 58 changes/sec × avg 5 alerts/product = 290 evaluations/sec — 1000× more efficient
>
> [!TIP]
> **🟢 Staff+** — the dedup contract — don't re-alert if the price hasn't crossed the threshold fresh. Track last_alert_sent_at per alert; require the price to recover above threshold and drop below again before re-alerting. Without this, a price hovering at the threshold triggers a new alert on every crawl cycle
#### Deep dive 3: TSDB design for price history charts
> [!CAUTION]
> **🔴 Weak** — store every price event in PostgreSQL, query by product_id and time range
>
> [!WARNING]
> **🟡 Strong** — InfluxDB or TimescaleDB — append-only, time-partitioned, columnar compression. But the real win is multi-resolution rollups: raw price events for all-time history, daily min/max/avg for chart display. 7-day view uses raw data; 90-day uses daily rollup; 3-year uses weekly rollup
>
> [!TIP]
> **🟢 Staff+** — pre-compute rollups with TimescaleDB continuous aggregates — they update automatically on every insert, so a 3-year chart query fetches 156 weekly rows instead of 26K raw events. 170× faster query with no visible chart difference at typical chart widths
_Why the deep dives connect to the scaling problem: "Data collection at scale, event-driven processing, and efficient time-series queries." Each deep dive solves one challenge._
Interview script
1. Collection-first script.
2. "Clarifying questions: are we tracking prices on a single retailer like Amazon, or multiple? And what's the primary data collection strategy — web crawling, browser extension, or both?"
3. "Good — primarily Amazon, extension plus crawler. Core features: track product prices, store history, trigger threshold alerts, display price charts. Out of scope: affiliate revenue, deal scoring."
4. "Collection strategy: Chrome extension covers products users actively browse — real-time, zero crawl cost. Crawler covers the long tail at lower frequency. Extension-first for the top 1M products; crawler for the rest."
5. "Storage: two databases. PostgreSQL for users, products, subscriptions — relational, low volume. InfluxDB or TimescaleDB for price history — append-only, queried by time range, compressed. Never use PostgreSQL for time-series price data at 500M products."
6. "Multi-resolution rollups: raw price events for all-time history. Daily min/max/avg for chart display. 7-day view uses raw. 90-day view uses daily rollup. 3-year view uses weekly rollup. Pre-computed by TimescaleDB continuous aggregates — query hits the rollup table directly."
7. "Alerts: event-driven via Kafka. Price change event → Alert Service checks watchlists for that product_id. At 58 changes/sec × avg 5 alerts/product = 290 evaluations/sec. 1000× cheaper than polling all 100M active alerts every 5 minutes."
8. "Trust but verify: extension data can be wrong — A/B test prices, regional variants. Fast-accept for display. Flag >50% price changes for priority crawler verification before storing permanently."
Whiteboard
```
+----------------------+
| Website / Chrome |
| Extension |
+----------+-----------+
|
v
+------------------+
| API Gateway |
| auth rate limit |
+---+----------+---+
| |
GET price hist | | POST subscription
v v
+----------------+ +-------------------+
| Price History | | Subscription |
| Service | | Service |
+-------+--------+ +---------+---------+
| |
v v
+---------------------+ +---------------------+
| Price DB | | Primary DB |
| time series prices | | users products subs |
+----------+----------+ +----------+----------+
| |
| new validated price |
v |
+-------------------+ |
| Kafka / Event Bus |<----------+
| price change evt |
+---------+---------+
|
v
+--------------------------+
| Notification Service |
| find matching subs |
+------------+-------------+
|
v
+----------------------+
| Email Provider |
+----------------------+
PRICE COLLECTION SIDE
+----------------------+ +----------------------+
| Chrome Extension | | Web Crawler Service |
| product page views | | selective crawling |
+----------+-----------+ +----------+-----------+
| |
v v
+-----------------------------------------------+
| Price Ingestion / Validation Service |
| trust but verify suspicious updates |
+-------------------+---------------------------+
|
valid price | write
v
+--------------+
| Price DB |
+------+--------+
|
| publish price changed
v
+--------------+
| Kafka / Bus |
+--------------+
OPTIONAL FAST VERIFICATION LOOP
suspicious extension update
|
v
+------------------------------+
| Verification Queue |
+--------------+---------------+
|
v
+------------------------------+
| Priority Crawler |
| checks Amazon quickly |
+--------------+---------------+
|
v
+------------------------------+
| Validation Service updates |
| trust score and final price |
+------------------------------+
```
The main idea is this. Extension plus crawler collect prices, validation decides what to trust, validated price changes go into the price database, and those changes produce events that drive notifications. Separately, the read path for charts stays simple and fast through the Price History Service querying the time series price store.
If you are drawing this in an interview, I would start with just three lanes. Client API read path, data collection path, and notification path. That keeps the whiteboard clean and makes the story easy to explain.
Upload: client gets a pre-signed S3 URL and uploads directly. Async transcode generates multiple resolutions. Fan-out: post event → Kafka → fan-out worker pushes post_id into follower Redis sorted sets. Celebrities: fan-out skipped, pulled at read time and merged.
Problem
Photo sharing at massive scale. Two hard problems: media pipeline (ingest → transcode → CDN) and feed generation (hybrid push/pull for celebrity accounts).
Failures
**Fan-out worker falls behind for a celebrity's post (50M followers)**
Millions of followers don't see the post for 10+ minutes. Feed appears stale.
_Fix:_ Celebrity accounts (> threshold followers) bypass fan-out entirely. At feed read time, pull celebrity post IDs from Cassandra and merge. Threshold is a config value, not code.
**Media transcode job fails for a specific format/codec**
Some users see a broken media preview. Reels content doesn't play on older devices.
_Fix:_ Idempotent transcode jobs: store transcode status per (media_id, resolution). Failed resolution re-queued automatically. Serve available resolutions while others process. Never block upload confirmation on transcode completion.
**CDN cache miss on a viral Reel**
First million requests all reach origin S3. S3 request cost spike. Origin latency degrades.
_Fix:_ Pre-warm CDN for content from verified/celebrity accounts. Use CDN cache-forward headers. Multi-CDN setup (primary + fallback) prevents single CDN saturation.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 2B users, 500M DAU, 100M photos/videos uploaded/day, 5B feed reads/day |
| Read QPS | 5B / 86400 ≈ 57,870 feed read QPS — served from Redis sorted sets |
| Write QPS | 100M uploads / 86400 ≈ 1,157 upload QPS. Fan-out: 1,157 × avg 500 followers = 578,500 feed writes/s |
| Storage | 100M uploads/day × avg 3 MB processed = 300 TB/day to S3. Over a year: 100 PB. |
| Cache math | Feed cache: 2B users × 500 post IDs × 8 bytes ≈ 8 TB Redis. Across Redis cluster with 100 nodes: 80 GB/node — fine. |
| Verdict | Media storage cost is the defining cost constraint. CDN is the largest spend. Fan-out write volume (578K/s) requires massive Kafka + worker fleet. |
Design decisions
**Upload to CDN origin vs. S3 + CDN**
→ Upload to S3, CDN fronts delivery
S3 is the durable store. CDN is the delivery layer. Never upload directly to CDN origin (CDN is not durable storage). Pre-signed S3 URL for direct upload (bypass app servers entirely).
_Revisit when:_ Some CDNs support direct upload (Cloudflare R2). Evaluate if CDN vendor provides both storage and delivery.
**Stories vs. Feed: different architectures**
→ Stories: 24hr TTL objects with different ranking. Feed: ranked by ML model.
Stories are ephemeral — Redis with 24hr TTL, no fan-out needed (pull on open). Feed is curated — ML ranking over pre-fetched candidates. Different content types need different storage and serving strategies.
_Revisit when:_ Stories could use the same fan-out infrastructure as feed if the architectures converge in product.
**Explore vs. Follow feed**
→ Separate serving stack for Explore (interest graph) vs. Follow feed (social graph)
Follow feed: fan-out from people you follow, Redis sorted sets. Explore: interest-based recommendations from the whole graph, ML model, no fan-out needed. Mixing these in one pipeline creates complexity.
_Revisit when:_ Separate teams, separate models, separate serving. Explore is essentially a recommendation system problem, not a social feed problem.
Follow-up Q&A
**How do you handle a 100GB video upload?**
TUS resumable protocol: chunk the video client-side (10MB chunks). Server accepts chunks independently, stores in S3 multipart. Upload can pause/resume. Server stitches chunks once all received, then enqueues transcode job. Client gets confirmation of receipt, not completion of transcode.
**How do you serve the right image resolution for different devices?**
Transcode generates multiple resolutions: thumbnail (150x150), feed (1080x1080), full (original). URL encodes resolution: cdn.instagram.com/media/{id}/1080.jpg. Client requests the appropriate resolution based on device screen density and connection speed.
**How does the ML recommendation ranking work?**
Two-stage retrieval + ranking: (1) candidate generation — retrieve top-1000 posts from social graph + interest graph, (2) ranking — ML model scores each candidate using user features, post features, interaction history. Serve top-50. This is a separate service from the feed delivery infrastructure.
**How do you handle copyright violations in user uploads?**
Perceptual hash (pHash) of every upload compared against a hash database of known copyrighted content. Match within threshold → quarantine for review. Audio fingerprinting for music in Reels (AcrCloud). DMCA takedown system for post-publish violations.
**How would you handle a data center outage?**
Multi-region active-active deployment. Cassandra replication across regions. S3 cross-region replication for media. Redis leader in primary region, read replicas in secondary. DNS failover (Route 53 health checks). Feed reads may be briefly stale — acceptable.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: end-to-end latency, consistency lag, fan-out depth. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: active-active or warm standby; conflict resolution on merge. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Single server, local disk storage, PG. Filters applied client-side. Upload + view. Handles 10K users.
**v2 — Scale** — S3 + CDN for media. Redis feed cache with fan-out. Pre-signed URL upload. Async transcode pipeline. Celebrity pull model. Handles 100M users.
**v3 — 2B users** — Multi-region. ML ranking for both Feed and Explore. Reels (video) as separate pipeline. Stories with 24hr TTL. Shopping integration. 100 PB/year media storage.
Why it's hard to scale
The hard part in Instagram is the feed. One new post can create a lot of downstream work, and one feed request can also be expensive if you build it on demand.
There are three main scaling pain points. First, feed generation has a fan-out problem. If you compute the feed at read time, one user request may need posts from hundreds or thousands of followed accounts, then merge and sort them fast. If you precompute feeds at write time, one new post may need to be pushed into millions of follower feeds. Second, media delivery is heavy. Photos and especially videos are large, so uploads, storage, and global low-latency delivery all get expensive fast. Third, load is very uneven. Most users are normal, but celebrity accounts create hot spots because one post can trigger huge write amplification and huge read traffic at the same time.
So the short interview answer is this. Instagram is hard to scale because it combines feed fan-out, massive media storage and delivery, and hot celebrity traffic. That is why a hybrid feed model is usually the best default. Precompute for normal users, then merge celebrity posts at read time.
Key points
- **Pre-signed URL upload** — Client uploads directly to S3. App server only issues the URL and records metadata.
- **Async parallel transcode** — S3 upload triggers a job. Workers for each resolution run in parallel.
- **CDN delivery** — Processed images served from CDN. Never serve media from app servers.
- **Fan-out on write** — Post event → Kafka → fan-out worker → push post_id into follower Redis sorted sets.
- **Celebrity threshold** — Accounts above a follower threshold get pull-at-read treatment.
> Pre-signed URL + async transcode for upload. Hybrid fan-out for feeds.
Tradeoffs
**Push fan-out vs pull for celebs** — Push to 50M+ follower caches is catastrophic write amplification. Pull celebrity posts at read time and merge with pre-built feed is the only viable approach for large accounts.
**Pre-signed URL vs proxy upload** — Pre-signed URL: client uploads directly to S3, app servers never touch the bytes. Proxy through app servers at 1,157 uploads/sec × avg 3MB = 3.5 GB/s through app tier — completely impractical.
**Transcode before vs after acknowledgment** — Blocking upload confirmation on full transcode (all resolutions) adds minutes of latency. Acknowledge on S3 receipt, transcode async, serve available resolutions progressively. Never block the user.
**CDN TTL vs cache purge on delete** — Long TTL maximizes CDN hit rate and reduces origin cost. But deleted posts must be purged immediately. Tag-based CDN purge (all variants of a post_id in one API call) reconciles both — long TTL by default, instant purge when needed.
> "Never push to 100M+ follower caches — catastrophic write amplification. Hybrid model is the critical staff-level insight."
Deep dives
#### Deep dive 1: Media upload pipeline — pre-signed URLs, async transcode, multiple resolutions
> [!CAUTION]
> **🔴 Weak** — accept upload through app servers, store to S3, transcode sequentially before acknowledging
>
> [!WARNING]
> **🟡 Strong** — the scaling pain is the read side of media: photos and videos are large, globally distributed, and must load fast. The write side has different constraints: uploads are infrequent, can tolerate latency, but must be resumable
>
> [!TIP]
> **🟢 Staff+** — (1) client requests a pre-signed S3 URL — app server never sees the file bytes. (2) Client uploads directly to S3 (bypasses all app servers — critical for cost and throughput). (3) S3 upload event triggers async transcode job (Kafka or Lambda). (4) Transcode workers process in parallel: thumbnail (150×150), feed (1080×1080), full resolution (original), HEVC-compressed video if applicable. (5) Processed outputs written to S3 behind CDN. Transcode workers can be right-sized: CPU-heavy but stateless, easy to scale horizontally. Staff+ failure mode: transcode job fails for one resolution. Store transcode status per (media_id, resolution). Failed resolutions are retried independently. Video is available at successfully-transcoded resolutions while others are processing. User never sees a blank feed card — serve the best available resolution
#### Deep dive 2: Feed generation — hybrid fan-out with celebrity threshold
_The fan-out problem is the defining system design challenge for Instagram. At 2B users with 500M DAU, a celebrity with 50M followers posting once generates 50M Redis write operations for that post_
> [!CAUTION]
> **🔴 Weak** — async workers handle it
>
> [!WARNING]
> **🟡 Strong** — tiered fan-out with explicit threshold. Normal accounts (< 1M followers): fan-out on write — post ID pushed to each follower's Redis sorted set via async Kafka workers. Celebrity accounts (> 1M followers): skip fan-out entirely. At feed read time: user's pre-built feed from Redis + latest N posts from each followed celebrity fetched from Cassandra, merged in-memory
>
> [!TIP]
> **🟢 Staff+** — the threshold is not binary — it's a function of follower count and current fan-out worker lag. Monitor lag: if fan-out workers fall behind (>2 min), dynamically lower the threshold to reduce load. The merge at read time: user follows 3 celebrities, each contributes last 20 posts = 60 post candidates + 980 from pre-built feed. Sort by timestamp, serve top 50. Merge cost: O(C × log C) where C is typically < 10, negligible
#### Deep dive 3: CDN strategy for media delivery at global scale
> [!CAUTION]
> **🔴 Weak** — put one CDN in front of S3, set Cache-Control headers, let it fill naturally
>
> [!WARNING]
> **🟡 Strong** — 99%+ of media reads are served from CDN — this is what makes Instagram's media delivery economically viable
>
> [!TIP]
> **🟢 Staff+** — CDN design: (1) Multi-CDN — use two CDN providers. Route requests to whichever CDN has lower P95 latency for that region (measured by synthetic probes). CDN failover in seconds. (2) Cache warming — for content from large accounts, pre-push content to CDN POPs (Points of Presence) in relevant regions before publication. (3) URL structure encodes resolution: cdn.instagram.com/media/{id}/1080.jpg — CDN can cache all resolutions independently. (4) Signed CDN URLs — prevent hotlinking and unauthorized access. URL includes HMAC signature with expiry. CDN validates signature at edge without origin call. (5) Cache invalidation — when a post is deleted, purge the CDN cache for all resolution variants. CDN APIs support tag-based purge: tag all media variants with the post_id, purge by tag on deletion
_Why the deep dives connect to the scaling problem: "Feed fan-out, media blob delivery, and celebrity hot spots." Each deep dive addresses one dimension._
Interview script
1. Media-pipeline-first script.
2. "Clarifying questions: are we designing just photos, or also Reels (video)? And what are the two core use cases — upload + feed read?"
3. "Good — photos plus short video. Core features: upload media, fan-out to followers, serve feed. Out of scope: Stories (different TTL model), Shopping, live streaming."
4. "Scale: 2B users, 500M DAU, 100M uploads/day, 5B feed reads/day. Two big numbers: 1,157 upload QPS and 578K fan-out writes/sec (avg 500 followers × 1,157 uploads/s). Fan-out is the dominant write load."
5. "Upload pipeline: client gets a pre-signed S3 URL — never proxy media bytes through app servers. Client uploads directly. S3 receipt triggers async transcode: thumbnail, 720p, 1080p in parallel. Video is available at 360p within seconds, higher resolutions follow. Never block upload confirmation on full transcode completion."
6. "Fan-out: new post → Kafka → async fan-out workers → write post_id to each follower's Redis sorted set. For accounts above ~1M followers, skip fan-out entirely. At feed read time, pull their recent posts from Cassandra and merge with the pre-built feed."
7. "CDN: all media reads served from CDN. 99%+ hit rate target. For large accounts: pre-warm CDN before publication. Signed CDN URLs with short expiry prevent hotlinking."
8. "Key tradeoff I'd name: the celebrity threshold for fan-out is a config value, not code. Tune it based on observed fan-out worker lag — if workers fall behind, lower the threshold dynamically."
Whiteboard
```
+-------------------+
| Mobile / Web |
| Clients |
+---------+---------+
|
v
+-------------------+
| API Gateway |
| auth rate limit |
+----+----+----+----+
| | |
POST /posts -----+ | +----- GET /feed
POST /follows ---------+
+-------------------+ +-------------------+ +-------------------+
| Post Service | | Follow Service | | Feed Service |
| create post meta | | follow unfollow | | read feed |
+----+---------+----+ +---------+---------+ +----+---------+----+
| | | | |
| | | | |
| v v | v
| +-----------+ +-----------+ | +------------+
| | Posts DB | | FollowsDB | | | Redis |
| | DynamoDB | | DynamoDB | | | feed zset |
| +-----------+ +-----------+ | | post cache |
| | +-----+------+
| | |
| | v
| | +------------+
| +-->| Posts DB |
| | BatchGet |
| +------------+
|
| presigned upload URL
v
+-------------------+ multipart upload +-------------------+
| Blob Storage |<------------------------------>| Client |
| S3 | +-------------------+
+---------+---------+
|
v
+-------------------+
| CDN |
| edge cache media |
+---------+---------+
|
v
+-------------------+
| Media Delivery |
| photos and videos |
+-------------------+
Async fanout path after new post
Post Service
|
v
+-------------------+
| Queue / Topic |
| new post events |
+---------+---------+
|
v
+-------------------+
| Feed Fanout Worker|
| async background |
+----+---------+----+
| |
| v
| +-----------+
| | FollowsDB |
| | followers |
| +-----------+
|
v
+-------------------+
| Redis |
| update feed:user |
+-------------------+
Celebrity hybrid read path
Feed Service
|
+----> Redis precomputed feed for normal accounts
|
+----> Posts DB for recent celebrity posts
|
v
merge by timestamp and return page
```
The mental model is two big flows. Write flow stores post metadata and media, then asynchronously updates follower feeds. Read flow pulls mostly from Redis, then hydrates post metadata from the posts store, with a hybrid read for celebrity accounts.
View events → Kafka → Flink aggregates per videoId using Count-Min Sketch for approximate frequency counting. Updates Redis sorted set via ZINCRBY. Top-K = ZREVRANGE. Lambda Architecture: daily Spark/ClickHouse batch reconciles exact counts.
Problem
Computing and serving the most-viewed videos in real-time at YouTube scale.
The challenge: exact counting at this scale is prohibitively expensive. The system must balance real-time approximation vs exact billing accuracy.
Failures
**Flink job falls behind on a viral video event (billion views in 1 hour)**
Top K list is stale. The viral video doesn't appear in trending for minutes.
_Fix:_ Flink auto-scales by input lag. Kafka partition count pre-sized for peak throughput. For extreme events: dedicated high-priority Kafka topic for top-1000 videos by view velocity.
**Redis sorted set for Top K becomes a hot key**
All writes (ZINCRBY) and all reads (ZREVRANGE) go to one Redis shard holding the top-K set. It becomes the bottleneck.
_Fix:_ Shard top-K by time window: one set per time window (top-K-hourly-2024-01-01-14, etc.). Flink writes to current window set. Query merges across windows. Local in-process cache for read-heavy consumers.
**Count-Min Sketch error rate too high for a specific video**
A mid-tier video's count is significantly over-estimated due to hash collisions. Appears in trending incorrectly.
_Fix:_ Increase sketch width (more hash functions) to reduce collision probability. For top-100 known videos, use exact counting in parallel (small set, cheap). Use sketch for the long tail only.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 2B users, 1B views/day, 800M videos, top-K query at global/country/category level |
| Read QPS | 1B views / 86400 ≈ 11,574 view events/s — into Kafka |
| Write QPS | Flink processes 11,574 events/s, emits ~1,000 ZINCRBY updates/s (only changed videos) |
| Storage | Count-Min Sketch: width=2000, depth=7 → 14,000 counters × 8 bytes = 112 KB per sketch. Tiny. Redis sorted set for Top K: K=1,000 × 8 bytes = 8 KB. Both trivially small. |
| Cache math | Top-K query: served from Redis sorted set, O(log N + K), sub-ms. 11,574 events/s fanout → Flink aggregation reduces to ~1,000 Redis writes/s. |
| Verdict | Event ingestion volume (11K/s) is manageable for Kafka. The key design insight is aggregation in Flink before writing to Redis — not writing per-view. |
Design decisions
**Lambda architecture (batch + stream) vs. stream only**
→ Lambda: Flink (speed layer) + Spark batch (batch layer)
Stream only is approximate. For ad revenue, creator monetization, and copyright detection, exact counts are legally required. Batch path reconciles. Staff-level answer: explicitly say 'Lambda Architecture' and explain why both paths are needed.
_Revisit when:_ Kappa architecture (stream only, replay for corrections) is an alternative if batch reconciliation latency is acceptable.
**Count-Min Sketch vs. exact counting**
→ Count-Min Sketch for long tail, exact counting for Top 1000
Exact counting for 800M videos would require 6.4 GB of counters in memory. CMS handles this in 112 KB with 1-5% error. Top 1000 exact counting is cheap (8 KB) and eliminates error for the videos that actually matter for trending.
_Revisit when:_ HyperLogLog for distinct viewer count (vs. total view count). Different data structure, different problem.
**Tumbling vs. sliding window for trending**
→ Tumbling windows (1hr, 24hr, 7day) for simplicity
Sliding windows give smoother trending signal but require keeping state for the full window duration. Tumbling windows are simpler: clear window at boundary, start fresh. Perceived smoothness difference is minimal for human-readable trending lists.
_Revisit when:_ Sliding windows with Flink's native windowing for smoother real-time trending signal.
Follow-up Q&A
**How do you prevent view count manipulation (bots)?**
Filter bot views at ingestion: IP rate limiting, user-agent validation, session token validation. Don't count views < 30s watch time. ML bot detection model. Views that pass validation go to Kafka. Views that fail: logged but not counted. Periodic batch audit of view patterns.
**How would you add a 'trending in your country' feature?**
Add country code to the Kafka event. Flink maintains per-country CMS and sorted set. Redis key: top-k:{country_code}:{window}. Adds linear memory cost per country tracked.
**How do you reconcile the batch and stream layers?**
Batch job (Spark on Hadoop/GCS) runs daily: exact count per video_id from raw event log. Writes exact counts to the serving layer (ClickHouse or BigQuery). Stream layer's approximate counts are used for real-time display. When batch completes, canonical count overwrites approximate count.
**What's the latency from view event to appearing in trending?**
P50: <2 minutes (Kafka ingestion + Flink window emit + Redis write + cache invalidation). This is the Flink tumbling window size. For breaking events: reduce window to 5 minutes at the cost of more Redis writes.
**How would you build this for a specific category (e.g., Top K Gaming videos)?**
Add category_id to Kafka event (requires video metadata join in Flink, or pre-tag at event time). Maintain per-category CMS + sorted set. Flink handles this with keyed streams: key by (category_id, window) instead of just (window).
**How do you handle duplicate view events from retries?**
Client-generated view_id UUID. Bloom filter or Redis SET at ingestion dedupes within 24h window. Exact dedup table for top creators where billing disputes matter.
**What happens when Flink falls behind during a viral video?**
Monitor consumer lag. Autoscale Flink task managers. Pre-provision extra Kafka partitions for peak events. Trending list may lag 2–5 min — show 'updating' indicator in UI.
**How do you store raw events vs aggregates cost-effectively?**
Hot raw events 7–30 days in Kafka/S3. Hourly/daily aggregates in ClickHouse forever. Query API routes by window size — never scan raw log for dashboard Top K.
Evolution
**v1 — Batch only** — Daily Spark job counts views from raw log. Top K served from ClickHouse. 24hr latency. Good enough for 'most viewed this week' but not 'trending now'.
**v2 — Stream layer (Lambda)** — Flink stream processing with CMS. Redis sorted set for real-time Top K. Batch layer remains for exact daily counts. Trending now with <2min latency.
**v3 — Global + personalized** — Per-country, per-category Top K. Personalized trending (blends global signal with user history). Bot detection in ingestion pipeline. Live view count display on video pages.
Why it's hard to scale
The hard part in YouTube Top K is that it looks like a simple ranking problem, but at scale it becomes a streaming aggregation problem. You are not just storing view counts. You are ingesting a huge firehose of views, updating counts fast enough, and still answering top K queries for different time windows with very low latency.
There are three main scaling pain points. First, write volume is massive. Every view is an event, so naive per view database updates fall over quickly. Second, windowed queries are expensive. Top K for the last hour, day, and month means you cannot just sort one static table. You need rolling aggregates over huge amounts of data. Third, precision plus low latency is a tough combo. If the result must be exact and returned in milliseconds, you usually need precomputation and caching, not on demand scans.
A fourth issue is cardinality. There are billions of videos, but only a tiny fraction belong in the top K. That means you need to process a very large universe of IDs just to find a very small answer set. So the interview summary is this. YouTube Top K is hard because it combines massive write throughput, expensive time window aggregation, and a need to precompute exact rankings fast enough to serve cheaply.
Key points
- **Lambda Architecture** — Real-time path for fast approximate results. Batch path for exact daily reconciliation. Name both paths explicitly.
- **Count-Min Sketch** — Approximate frequency in O(1) space with 1–5% error. Massive space savings vs exact counting.
- **Flink tumbling windows** — Aggregate view counts per videoId in fixed time windows.
- **Watermarking** — Flink watermark with 1-minute allowed lateness. Handles late events.
- **Redis sorted set for Top-K** — ZINCRBY to update scores, ZREVRANGE to read Top-K.
> Staff expectation: name Lambda Architecture explicitly with both paths.
Tradeoffs
**Exact counting vs Count-Min Sketch** — Exact counting for 800M videos = 6.4 GB of counters updated at 11K/sec. Count-Min Sketch handles this in 152 KB with 1–5% error — acceptable for a trending list, not for billing.
**Real-time only (Kappa) vs Lambda Architecture** — Real-time alone is approximate — insufficient for creator revenue and copyright. Lambda adds exact batch reconciliation at the cost of two pipelines. For ad revenue, Lambda is non-negotiable.
**Tumbling vs sliding windows for trending** — Tumbling windows (1hr, 24hr) are simpler — state resets at boundary. Sliding windows give smoother trending signal but require keeping state for the full window duration. Tumbling is the right default.
**Kafka partition by video_id vs random** — Partition by video_id ensures all events for one video go to one Flink operator — correct aggregation without cross-partition coordination. Random partitioning requires a shuffle step to aggregate per video.
> "Count-Min Sketch trades a small amount of accuracy for massive reduction in space and compute. Approximate is good enough for trending. The batch path gives exact counts when needed."
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
#### Deep dive 1: Count-Min Sketch — approximate frequency counting at stream scale
> [!CAUTION]
> **🔴 Weak** — maintain a HashMap of video_id → count, increment on every view event. At 800M videos × 8 bytes = 6.4 GB of counters updated at 11K/sec — too expensive, and one hot video creates a write bottleneck
>
> [!WARNING]
> **🟡 Strong** — Count-Min Sketch — a 2D array of W×D counters. On each event: hash video_id with D hash functions, increment the counter at each position. To estimate count: take the minimum of the D values
>
> [!TIP]
> **🟢 Staff+** — at 1% error with 99.9% confidence, W ≈ 2718, D ≈ 7. Total memory: 152 KB per time window vs 6.4 GB for exact counting. For Top K specifically: combine CMS with a Min-Heap of K items — only track candidates whose estimated count exceeds the current K-th largest. The heap has K entries; the sketch stays constant size regardless of cardinality
#### Deep dive 2: Lambda Architecture — why both stream and batch paths are needed
> [!CAUTION]
> **🔴 Weak** — use Flink stream processing only — it handles the volume
>
> [!WARNING]
> **🟡 Strong** — stream-only (Kappa architecture) gives approximate results. For ad revenue, creator monetization, and copyright detection, approximate counts are legally insufficient. Lambda Architecture: real-time path gives approximate counts for the trending dashboard (fast, approximate), batch path gives exact counts for billing and reporting (slow, exact)
>
> [!TIP]
> **🟢 Staff+** — implementation: real-time path (Flink + CMS + Redis sorted set) runs continuously. Batch path (Spark on S3 event lake) runs daily, produces audited exact counts. Serving layer stores both: consumers use exact counts when available, approximate otherwise. Never use approximate counts for financial reporting
#### Deep dive 3: Time window management — tumbling vs. sliding, late event handling
> [!CAUTION]
> **🔴 Weak** — use a single global counter per video — no time window awareness
>
> [!WARNING]
> **🟡 Strong** — separate state per window (1hr, 24hr, 7day) using Flink tumbling windows. Each window starts fresh at its boundary
>
> [!TIP]
> **🟢 Staff+** — tradeoff: tumbling windows are simpler but the trending list jumps at boundaries — a video popular for the last 59 minutes drops off suddenly at the 1-hour mark. Flink sliding windows give smoother signal but require keeping state for the full window duration. Recommendation: tumbling for longer periods (24h, 7d) where boundary jumps are acceptable, sliding for the 1-hour list where users expect smooth changes. Late event handling: Flink watermark with 2-minute allowed lateness. Events arriving later go to the batch path for exact reconciliation
_Why the deep dives connect to the scaling problem: "Massive write throughput, windowed aggregation, and precision vs. latency." Each deep dive addresses one constraint._
Interview script
1. Lambda Architecture framing.
2. "I'd design this as a Lambda Architecture: a real-time streaming path for fast approximate Top-K, and a batch path for exact reconciliation."
3. "Real-time path: view events → Kafka partitioned by video_id → Flink streaming job → Count-Min Sketch for approximate frequencies → update Redis sorted set via ZINCRBY → Top-K = ZREVRANGE."
4. "Count-Min Sketch gives approximate frequency counts with 1–5% error using a fraction of the memory of exact counting. For a trending video list, that's good enough."
5. "Batch path: raw events land in ClickHouse. A daily Spark job computes exact view counts and reconciles against the real-time Redis counts. Exact numbers for reporting and billing."
6. "I'd name this Lambda Architecture explicitly — real-time for speed, batch for correctness. Both are needed."
Whiteboard
```
+----------------------+
| YouTube Clients |
+----------+-----------+
|
| watch events
v
+----------------------+
| Video Serving System |
+----------+-----------+
|
| publish ViewEvent(videoId, ts)
v
+---------------------+
| Kafka ViewEvent |
| topic partitioned |
| by videoId |
+----+----+----+-----+
| | |
consume | | | consume
v v v
+----------------------------------+
| Flink / Stream Aggregator |
| watermark for late events |
| minute or hour tumbling windows |
| count views per video |
+----------------+-----------------+
|
batched aggregates per shard
|
+------------------------+------------------------+
| | |
v v v
+--------------+ +--------------+ +--------------+
| Views DB S1 | | Views DB S2 | ... | Views DB SN |
| shard by | | shard by | | shard by |
| videoId | | videoId | | videoId |
+------+-------+ +------+-------+ +------+-------+
| | |
| keep window tables | keep window tables |
| | |
| - all_time | - last_hour |
| - last_day | - last_month |
v v v
+---------------------------------------------------------------+
| indexed aggregate tables per shard |
| query top K locally on each shard |
+---------------------------+-----------------------------------+
|
| periodic fanout query
v
+-------------------------------+
| Top K Precompute Job / Cron |
| query each shard for local K |
| merge into global top K |
+---------------+---------------+
|
| write precomputed results
v
+--------------------+
| Redis Cache |
| top-k:last_hour |
| top-k:last_day |
| top-k:last_month |
| top-k:all_time |
+---------+----------+
|
v
+----------------------------+
| Top K API Service |
| GET /views/top-k?window&k |
+-------------+--------------+
|
v
+------------------+
| Load Balancer |
+--------+---------+
|
v
+------------------+
| Clients |
+------------------+
```
If you are presenting this in an interview, the clean story is this. Kafka absorbs the firehose of view events. Flink batches and aggregates views by video for a time bucket. Sharded databases store pre-aggregated counts for each window. A precompute job pulls local top K from each shard, merges them, and writes the final answers into Redis. The API just reads from Redis, which is how you hit the tens of milliseconds latency target.
If you want, I can also give you a simpler interview version with only 6 boxes so it is easier to draw under time pressure.
Uber — ride-sharing Hard
Geo matching · Location tracking · Trip state machine
Redis GEOPostgreSQLWebSocketGeohashTrip state machine
Driver location → Redis GEOADD every 4 seconds. This is ephemeral — never written to PostgreSQL. Rider requests → GEORADIUS finds nearby drivers → rank → dispatch. Trip state machine in PostgreSQL: REQUESTED → MATCHED → IN_PROGRESS → COMPLETED.
Problem
Matching riders with drivers in real time, tracking trips, and calculating pricing dynamically.
The hard parts: driver location is a high-frequency ephemeral stream (never the DB), geospatial matching must be fast, and the trip must be durably recorded.
Failures
**Redis GEO data is stale when drivers stop sending location updates**
Matching service finds nearby drivers who are actually offline or far away. Dispatching fails, rider waits.
_Fix:_ Driver location has a TTL in Redis (e.g., 15 seconds). Expired entries automatically removed from the geo index. Driver app sends heartbeat even when idle. If location key expires, driver is considered offline.
**Matching service assigns the same driver to two riders simultaneously**
Double booking. Driver gets two pickup requests. One rider is stranded.
_Fix:_ Driver state machine: AVAILABLE → DISPATCHED (atomic in Redis). Use SETNX driver:{id}:state = DISPATCHED. First match wins. Second match sees DISPATCHED status and picks the next available driver.
**City-level demand spike (concert ending)**
GEORADIUS returns no available drivers in a 2km radius. Demand/supply ratio spikes 10×.
_Fix:_ Expand search radius dynamically when no drivers found. Notify nearby idle drivers. Surge pricing activation triggers driver supply response. Circuit breaker: stop new requests if matching latency exceeds 5 seconds.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 10M drivers active at peak, location update every 4s, 5M ride requests/day |
| Read QPS | GEORADIUS queries: 5M requests / 86400 ≈ 58 matching QPS — low, surprisingly |
| Write QPS | 10M drivers × 1 update / 4s = 2.5M GEOADD writes/s — this is the scaling challenge |
| Storage | Driver location in Redis: 10M entries × 50 bytes ≈ 500 MB. Trips in PG: 5M/day × 500 bytes × 365 ≈ 914 GB/year. |
| Cache math | 2.5M GEOADD/s is the number. Redis handles ~500K ops/s per node → need 5 Redis nodes just for location writes. Or: batch updates per driver to 1/8s (still fresh enough for matching). |
| Verdict | Location write throughput (2.5M/s) is the actual scaling challenge. Not ride requests (58 QPS). Every optimization discussion should start here. |
Design decisions
**Redis GEO vs. PostGIS vs. custom geospatial index**
→ Redis GEO (GEOADD / GEORADIUS)
PostGIS at 2.5M writes/s is impossible — it's a relational DB. Custom geohash grid in Redis is equivalent to what Redis GEO does internally. Redis GEO is built-in, battle-tested, and handles our write volume with horizontal sharding.
_Revisit when:_ H3 hexagonal indexing (Uber's actual approach) gives more uniform cell sizes than geohash and handles region boundaries more cleanly.
**Trip state in PG vs. distributed state machine**
→ PostgreSQL with state machine transitions
Trip state changes are infrequent (one per minute per trip) and require ACID guarantees (billing, receipts). PG at 58 QPS is trivial. Distributed state machine (Temporal, Conductor) adds complexity not needed at this scale.
_Revisit when:_ Temporal workflow engine for complex multi-step trip flows (shared rides, multi-leg trips) where saga pattern is needed.
**Surge pricing: reactive vs. predictive**
→ Reactive (demand/supply ratio) with ML prediction overlay
Pure reactive surge leads to oscillation (surge → drivers flood in → surge drops → drivers leave → surge returns). ML prediction smooths this: forecast demand 15 minutes ahead, pre-surge before the spike.
_Revisit when:_ Pure reactive is simpler. Add ML prediction only when driver supply oscillation becomes a measurable problem.
Follow-up Q&A
**How do you handle driver location privacy?**
Driver location is visible to matched rider only (not general public). Location is ephemeral in Redis — never persisted to PG in raw form. Trip GPS logs are retained for dispute resolution but not accessible in real-time by anyone except matched rider/driver.
**How does ETA calculation work?**
ETA = routing API call (Google Maps / Mapbox / internal routing). Inputs: driver current location + destination. Returns route + ETA. Cached per (driver_location_geohash, destination_geohash) with 1-min TTL. Never calculated from raw Euclidean distance — road network matters.
**How would you handle simultaneous ride requests from 10,000 users in one city?**
Matching is per-request (not batched). Each request does a GEORADIUS query independently. Redis GEO handles concurrent reads trivially. The bottleneck is driver availability, not the matching system. At 10K simultaneous requests in one area, surge pricing reduces demand while the driver fleet adjusts.
**How do you handle the vehicle type selection (UberX vs. UberXL vs. Black)?**
Redis GEO index is per vehicle type. GEORADIUS query targets the correct type index. Driver registration sets their vehicle type, which determines which geo index they're added to. Adding a vehicle type adds one more Redis GEO set — linear complexity.
**What happens if the rider cancels after the driver is dispatched?**
Trip state machine: DISPATCHED → CANCELLED. Driver state: DISPATCHED → AVAILABLE (immediate). Kafka event triggers: cancel notification to driver, cancel fee calculation (if within grace period), driver geo index re-add. Idempotent — driver can receive cancel event more than once safely.
**How do you implement shared rides (UberPool)?**
Matching becomes a combinatorial problem: find driver + route that accommodates multiple riders with minimal detour. Store active pool trips with ordered pickup/dropoff waypoints. Re-match only between trips, not mid-ride unless rider cancels.
**How do you shard location data globally?**
Shard Redis GEO by city/region geohash prefix. Cross-region matching only for airport/long-distance product modes. Each shard owns its driver supply pool.
**How do you recover if matching service crashes mid-dispatch?**
Trip row created with status MATCHING and driver_id candidate. On retry, idempotent matching reads existing trip — re-confirms same driver if still DISPATCHED, otherwise picks next. Never double-dispatch same trip_id.
Evolution
**v1 — MVP** — PG with PostGIS for driver location. Polling every 30s. 500ms matching. Works for 1 city with 1000 drivers. Breaks at any real load.
**v2 — Redis GEO** — Redis GEO for driver location (ephemeral). PG for trip state machine. WebSocket for live driver tracking. Basic surge pricing. Handles 10M drivers globally.
**v3 — Prediction + scale** — ML demand prediction for proactive surge. H3 hexagonal geo indexing. Multi-modal (Express Pool, Delivery). Real-time ETA with road network. Driver incentive system.
Why it's hard to scale
The hard part in Uber is not storing rides. It is matching the right driver to the right rider very fast while the whole system is constantly changing.
There are three big scaling pain points. First, driver location is a huge real time write stream. If millions of drivers send updates every few seconds, a normal database gets overwhelmed, and proximity search on raw latitude and longitude is too slow. Second, matching needs low latency and strong enough consistency. You cannot assign the same driver to two riders at once, so the system needs some kind of lock or reservation while still moving quickly. Third, demand is bursty and local. A concert ending can create a massive spike in one neighborhood, so even if global traffic looks fine, one region can become a hotspot.
A good interview summary is this. Uber is hard because it combines real time geospatial search, hotspot traffic, and correctness during matching. You are not just finding a nearby driver. You are doing it from fast moving data, under heavy local spikes, without double booking drivers.
Key points
- **Scope it first** — Core: rider requests ride, match to nearby driver, track driver location, complete trip, compute fare. Out of scope unless asked: surge pricing algorithm details, driver incentives, scheduled rides.
- **Driver location — Redis GEO only** — 2.5M GEOADD writes/sec. Redis handles this; PostgreSQL collapses at ~50K writes/sec. Location is ephemeral: EXPIRE on every key (15s TTL). Expired = driver offline. Never persist raw location to PostgreSQL.
- **GEORADIUS for matching** — GEORADIUS drivers:available LONGITUDE LATITUDE 5 km ASC COUNT 10. Returns nearest available drivers. Filter by AVAILABLE state (SETNX driver:{id}:status=DISPATCHED — atomic dispatch).
- **Trip state machine — PostgreSQL** — REQUESTED → MATCHED → DRIVER_EN_ROUTE → IN_PROGRESS → COMPLETED. Each transition is an immutable event. PostgreSQL for ACID durability — the trip is the financial record.
- **Surge pricing — geohash cells** — Divide city into geohash cells (~1 km²). Compute demand/supply ratio per cell every 2 min. Surge multiplier = f(ratio). ML demand prediction pre-surges 15 min ahead to prevent oscillation.
- **Driver state machine — prevent double-dispatch** — AVAILABLE → DISPATCHED: SETNX driver:{id}:status = DISPATCHED. First matcher wins. Second matcher finds key exists → pick next driver. TTL on the key prevents stuck state if matcher crashes.
- **Failure mode to name** — Matching service assigns driver but crashes before confirming to rider: driver gets a ping, rider sees no driver. On client retry, matching service finds driver already DISPATCHED to this trip (idempotency via trip_id) — re-confirms without re-dispatching.
> The staff-level key: driver location is ephemeral (Redis only). PostgreSQL is trip durability only.
Tradeoffs
**Redis for driver location vs PostgreSQL** — Redis handles 2.5M GEOADD writes/sec. PostgreSQL at that write rate collapses. Redis is the only correct choice for ephemeral location data — never write driver location to PG on every update.
**GEORADIUS vs H3 hexagonal indexing** — Redis GEORADIUS is built-in and works well. H3 hexagonal cells give more uniform area coverage and cleaner region boundaries. H3 is Uber's production choice but adds implementation complexity — GEORADIUS is the right interview answer.
**Surge pricing reactive vs predictive** — Pure reactive surge creates oscillation — prices spike, drivers flood in, prices drop, drivers leave. ML demand prediction (15-min lookahead) pre-surges before the spike and smooths driver supply. Tradeoff: model infrastructure required.
**Pull matching vs push dispatch** — Pull: workers poll Redis for ride requests. Push: matching service dispatches directly to driver app. Push is lower latency (driver gets notified immediately) but requires reliable push delivery. Pull is simpler but adds polling overhead.
> "Driver location = Redis, trip = PostgreSQL. Never write ephemeral high-frequency data to a relational database."
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
#### Deep dive 1: Driver location — ephemeral Redis GEO at 2.5M writes/second
> [!CAUTION]
> **🔴 Weak** — write driver location to PostgreSQL on every update — it's the source of truth. At 10M drivers × 1 update/4s = 2.5M writes/sec, PostgreSQL collapses at ~50K writes/sec
>
> [!WARNING]
> **🟡 Strong** — Redis GEO (GEOADD) for location — in-memory, O(1) write, supports GEORADIUS queries natively. Set a 15-second TTL on every driver key: if the driver stops sending updates, they're automatically removed from the geo index
>
> [!TIP]
> **🟢 Staff+** — never write ephemeral high-frequency data to a relational database. Location is ephemeral — it changes every 4 seconds and is only needed for the current moment. PostgreSQL is for durable trip records, not live location pings. This distinction drives the entire architecture
#### Deep dive 2: Matching — low-latency GEORADIUS with driver state management
> [!CAUTION]
> **🔴 Weak** — query Redis for nearby drivers, pick the closest, dispatch
>
> [!WARNING]
> **🟡 Strong** — GEORADIUS returns candidates but dispatch requires an atomic state check — the same driver cannot be assigned to two riders simultaneously. SETNX driver:{id}:status = DISPATCHED: first matcher wins, second finds the key already set and picks the next available driver. TTL on the key prevents stuck DISPATCHED state if the matching service crashes mid-dispatch
>
> [!TIP]
> **🟢 Staff+** — : matching service assigns a driver but crashes before confirming to the rider. On client retry, the matching service finds the driver already DISPATCHED to this trip (idempotency via trip_id) and re-confirms without re-dispatching. The trip_id is the idempotency key, not the driver_id
#### Deep dive 3: Trip state machine and financial integrity
> [!CAUTION]
> **🔴 Weak** — store current trip status in Redis for fast access
>
> [!WARNING]
> **🟡 Strong** — PostgreSQL for all trip state — REQUESTED → MATCHED → DRIVER_EN_ROUTE → ARRIVED → IN_PROGRESS → COMPLETED. Each transition uses optimistic locking (UPDATE trips SET status=? WHERE id=? AND status=? AND version=N). Zero-row update means a concurrent transition happened — retry or surface the conflict
>
> [!TIP]
> **🟢 Staff+** — the trip is the financial record. It must be durable, auditable, and ACID. Event sourcing for the trip: every state transition is an immutable event row. The current state is derived from the event log. This gives a complete audit trail for fare disputes and is required for financial regulatory compliance
_Why the deep dives connect to the scaling problem: "Real-time geospatial search, hotspot traffic, correctness during matching." Each deep dive addresses one layer._
Interview script
1. Separation-of-concerns script.
2. "Before I start: are we designing the full platform — rider, driver, matching, payments — or focusing on the core matching and location flow?"
3. "Good — core flow. Functional requirements: rider requests ride, system matches to a nearby driver, both track each other live, trip completes and fare is calculated. Out of scope: surge pricing algorithm internals, driver incentives."
4. "Scale: 10M drivers active at peak, location update every 4 seconds. That's 2.5M writes per second just for location — this is the number that drives the entire architecture."
5. "The core insight I'd lead with: separate ephemeral driver location from durable trip state. These have completely different consistency and storage requirements."
6. "Driver location: Redis GEO with GEOADD. TTL of 15 seconds — if a driver stops sending updates, they're automatically removed from the index. Never write location to PostgreSQL. 2.5M writes/sec would destroy a relational database."
7. "Matching: GEORADIUS returns nearby drivers. Dispatch atomically using SETNX on the driver's status key — first matcher wins. Second matcher finds the key already set and picks the next available driver."
8. "Trip state machine in PostgreSQL: REQUESTED → MATCHED → DRIVER_EN_ROUTE → IN_PROGRESS → COMPLETED. Each transition is a row in the event log. PostgreSQL gives ACID durability — the trip is the financial record."
9. "Live tracking: after matching, rider's app opens a WebSocket. Driver location updates flow from Redis through the delivery service to the rider's connection every 4 seconds."
Whiteboard
```
+-------------------+
| Rider Client |
| iOS / Android App |
+---------+---------+
|
v
+------+------+
| API Gateway |
| auth, rate |
| limiting |
+------+------+
|
+-------------+-------------+
| |
v v
+-------+--------+ +-------+--------+
| Ride Service | | Notification |
| fares, rides, |<-------->| Service |
| ride state | | push to driver |
+---+--------+---+ +-------+--------+
| | |
| | v
| | +------+------+
| | | Driver |
| | | Client |
| | +------+------+
| | |
| | v
| | PATCH /rides/{id}
| |
| +------------------------------+
| |
v v
+-------+--------+ +---------+---------+
| Ride DB | | Fare DB |
| rides, status, | | estimate records |
| rider, driver | +-------------------+
+----------------+
Fare estimate path
------------------
Rider Client -> API Gateway -> Ride Service -> Maps API
|
v
Fare DB
|
v
Fare response
Matching and location path
--------------------------
+-------------------+ POST /drivers/location +----------------------+
| Driver Client | ---------------------------------> | Location Service |
| GPS updates | | ingest driver coords |
+-------------------+ +----------+-----------+
|
v
+--------+---------+
| Redis Geo Store |
| current driver |
| locations |
+--------+---------+
|
v
+--------+----------+
| Matching Service |
| find nearest |
| available driver |
+---+-----------+---+
| |
| v
| +-----+------+
| | Redis Lock |
| | driver TTL |
| +-----+------+
| |
v v
+------+-----------+------+
| Ride DB update ride |
| requested or accepted |
+-------------------------+
Request ride flow
-----------------
```
Robinhood — stock trading Hard
Financial integrity · Event sourcing · Market data
Market data flows exchange → Kafka → Redis price cache. Order placement: client sends an idempotency key → API checks key in DB → one ACID transaction reserves buying power and appends an immutable event log entry → calls Broker API.
Problem
A commission-free stock trading platform. Real-time market prices, place orders, and portfolio accuracy.
Hard parts: idempotency (retries must not produce double trades), financial integrity (double-entry accounting), and event sourcing for audit trail.
Failures
**Broker API call succeeds but Robinhood server crashes before writing confirmation**
Order placed at exchange but no record in Robinhood's system. Customer's position is wrong. Money at risk.
_Fix:_ Two-phase: write 'pending' order to PG before calling broker. If broker succeeds, update to 'filled'. If Robinhood crashes after broker success: on restart, reconciliation job queries broker for pending orders and updates state. Idempotency key prevents duplicate broker calls.
**Market data Redis cache lags during extreme volatility (circuit breaker event)**
Customers see stale prices. Execute orders at wrong expected prices.
_Fix:_ Price feed has a freshness TTL — if price hasn't updated in 500ms, mark it as stale and show staleness indicator to user. Don't let customers execute orders on prices > 2s old without explicit acknowledgment.
**Order book for a hot stock becomes a hot key in the system**
GME squeeze: millions of users watching and placing orders for the same ticker. Single Redis key for GME price is hammered.
_Fix:_ Ticker-level read sharding: multiple Redis nodes hold the same key, consistent-hash read routing. Write goes to one master, fan-out to replicas. For options chain data (larger): separate read-through cache with CDN for static expirations.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 15M users, 1M DAU, 5 trades/day per active user, 10K price updates/s from exchange |
| Read QPS | 1M users × 20 price checks/day / 86400 ≈ 231 price check QPS — trivial, Redis handles |
| Write QPS | 1M × 5 / 86400 ≈ 58 order writes/s — tiny. Exchange feed: 10K price updates/s to Redis. |
| Storage | Event log: 58 events/s × 300 bytes × 86400 × 365 ≈ 600 GB/year. PG sharded by user_id after 1TB. |
| Cache math | Price cache: 10K active tickers × 100 bytes = 1 MB Redis. Trivial. Options chain cache: 10K tickers × 1KB = 10 MB. Still trivial. |
| Verdict | Financial correctness (not throughput) is the design constraint. 58 orders/s is easy. The hard part is the 2-phase commit with the exchange and the reconciliation job. |
Design decisions
**Event sourcing vs. mutable balance**
→ Event sourcing with materialized view for balance
Mutable balance: update one field. Simple. But you lose the audit trail — can't reconstruct 'what happened'. Event sourcing: every debit/credit is a row. Balance = sum(events) or maintained as a materialized view updated per-event. Legal and regulatory requirement for financial systems.
_Revisit when:_ Never revert this decision for a financial system. Auditability is non-negotiable.
**Synchronous vs. asynchronous broker API calls**
→ Synchronous with timeout + async reconciliation
Market orders should feel instant (< 500ms feedback). But broker API may be slow. Synchronous with 2s timeout: if timeout, mark order as PENDING, reconcile later. User sees 'order submitted' immediately, confirmation comes async.
_Revisit when:_ Pure async queue for all orders if broker latency is consistently > 500ms.
**Cash equity vs. margin account architecture**
→ Separate account types with separate collateral logic
Cash account: simple debit/credit. Margin account: buying power = cash + margin credit × leverage. Complex collateral math. These are different enough to warrant separate services with shared event log.
_Revisit when:_ Start with cash-only. Add margin as a separate service on top of the same event sourcing infrastructure.
Follow-up Q&A
**How do you handle a market halt (trading suspended on an exchange)?**
Market status comes from exchange data feed. When halt received: set market_status = HALTED for that exchange in Redis. Order service rejects new orders with 'market halted' error. Existing open orders: cancel or hold based on halt type (circuit breaker vs. halted for news). Resume when market_status = OPEN.
**How do you handle fractional shares?**
Store shares as integer (millionths of a share). 1 share = 1,000,000 units. All arithmetic in integer math — no floating point for financial values (never use float for money). Display layer divides by 1,000,000 for human-readable output.
**How would you add options trading?**
Options are separate instruments with expiration dates and strike prices. New instruments table. Options chain cache (all strikes for a ticker × all expirations) is larger than equity price cache — pre-fetch and cache with options-specific TTL (changes with time decay). Order book logic same as equity but with additional validation (sufficient collateral for selling options).
**How do you ensure tax lot accounting (FIFO/LIFO) for sells?**
Event log already has all buy events with purchase price and date. On sell: query buy events for this ticker in FIFO order, determine cost basis per lot sold. This is a read operation over the event log — expensive but only triggered on sell. Results cached in a tax lot materialized view.
**What happens during a market close → open gap (overnight price change)?**
Last price cached in Redis with a 'market closed' flag. During pre-market and post-market, update with AH/PM prices but show clearly as AH/PM prices (different from regular market hours). On market open, switch to real-time feed. No functional change to the architecture — just metadata on the price entry.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: end-to-end latency, consistency lag, fan-out depth. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: active-active or warm standby; conflict resolution on merge. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — PG for everything. Synchronous broker API. Simple balance table. Handles early users. Would fail audit and at any real trading volume.
**v2 — Financial correctness** — Event sourcing for ledger. Idempotency keys. 2-phase order flow. Reconciliation job. Redis price cache. Separate market data pipeline.
**v3 — Scale + products** — Options trading. Margin accounts. Fractional shares. Tax lot accounting. Order routing (multiple brokers). Extended hours trading.
Why it's hard to scale
The hard part in Robinhood is that it combines real-time market data with high-stakes correctness. You need prices to update fast, but you also need orders and cancels to be reflected accurately because a stale or lost update can cost real money.
There are three main scaling pain points. First, live price fan-out is big. Many users may watch the same symbol at once, so you do not want every client talking to the exchange directly. You usually centralize exchange connections, ingest the feed once, then fan updates out internally to many app servers. Second, order handling is latency sensitive and consistency sensitive at the same time. A user expects a buy or cancel to happen quickly, but you also need durable local state so you can recover if your system talks to the exchange and then crashes mid-flow. Third, the exchange is an external dependency, which makes everything harder. You have limited connections, limited request patterns, and partial failure cases where your database and the exchange can get out of sync.
A good mental model is this. Robinhood is hard because it is part realtime system and part financial workflow engine. The realtime side is about efficiently broadcasting shared price updates. The harder side is making sure order state stays correct across your system and the exchange, even when failures happen.
Key points
- **Scope it first** — Core: view portfolio, get real-time quotes, place market/limit orders, view order history. Out of scope unless asked: options, margin, crypto, tax lot optimization, fractional shares (unless asked).
- **Idempotency key — store before PSP call** — Client generates UUID. Server stores idempotency_key (status=PENDING) BEFORE calling broker. On retry: find PENDING key → query broker for status → update to COMPLETED or FAILED. Sequence prevents double orders.
- **Event sourcing for ledger** — Balance is never stored as a mutable field. Every debit/credit is an immutable event row. Balance = sum(events) materialized by a checkpoint. Required for regulatory compliance — 7-year immutable audit trail.
- **Market data — separate pipeline** — 10K price updates/sec from exchange → Kafka → Redis. App servers subscribe by ticker. Fan-out to users watching that ticker via WebSocket. High-frequency, approximate-OK (1-2s stale is fine for display).
- **Two-phase order execution** — (1) Reserve buying power in PG ledger (ACID). (2) Call broker API with idempotency key. (3) On broker success: commit debit + record fill. On broker timeout: mark PENDING, reconcile async. Never debit before broker confirmation.
- **Fractional shares — integers only** — Store shares as integer units (1 share = 1,000,000 units). All arithmetic stays exact. IEEE 754 float accumulates rounding errors across millions of transactions. Display layer divides by 1M.
- **Failure mode to name** — Broker API times out — did the order go through? Query broker using the idempotency key before retrying. Idempotency key tells the broker this is a retry, not a new order. Never assume timeout = failure for financial operations.
> Mental model: idempotency key prevents double charges. Event sourcing gives immutable audit trail.
Tradeoffs
**Event sourcing vs mutable balance** — Event sourcing = immutable audit trail, temporal queries, full history — required for financial regulatory compliance. Mutable balance is simpler but loses the audit trail. No real choice for a financial system.
**Market data in Redis vs DB** — Redis handles thousands of price updates/sec easily. DB would be overwhelmed. Redis is cache — source of truth is the exchange feed. On Redis failure, re-seed from the feed.
**Synchronous vs async broker API call** — Synchronous with 2s timeout gives users immediate feedback. If timeout occurs: mark PENDING, reconcile async. Pure async queue adds latency for the common case (PSP responds in <500ms). Sync with fallback is the right default.
**Fractional shares as float vs integer** — Storing share quantities as IEEE 754 float introduces rounding errors that compound across millions of transactions. Store as integer (millionths of a share: 1 share = 1,000,000 units). All arithmetic stays exact. Display layer divides.
> "Idempotency key is the #1 concept. Without it, network retries = double orders. Event sourcing = immutable audit log you can't corrupt."
Deep dives
#### Deep dive 1: Idempotency — preventing duplicate orders under network retries
> [!CAUTION]
> **🔴 Weak** — check if a similar order exists before placing a new one
>
> [!WARNING]
> **🟡 Strong** — the hardest financial correctness problem: a network timeout means the client doesn't know if the order went through. If the client retries, it might place two orders. At $50 average trade value and 58 orders/second, a 1% duplicate rate = $29,000/second in double-charges. Unacceptable
>
> [!TIP]
> **🟢 Staff+** — design: client-generated idempotency key (UUID v4) included in every order request. Server logic: (1) check if idempotency_key exists in DB — if yes, return the cached response (no new order created). (2) If no, proceed: insert idempotency_key record + create order in one ACID transaction. On broker API call: include the idempotency_key as the broker's own idempotency parameter (Stripe, Alpaca, etc. support this). If broker returns success + idempotency key matches: return cached success. If broker returns error: return cached error. The idempotency record must be stored BEFORE the broker API call — if stored after, a crash between broker success and DB write results in a duplicate on retry. Timing: store idempotency key as PENDING, call broker, update to COMPLETED/FAILED. On retry: if PENDING state found, check broker directly (re-query by the same idempotency key)
#### Deep dive 2: Event sourcing — the immutable ledger for financial compliance
> [!CAUTION]
> **🔴 Weak** — maintain a balance column, UPDATE on every debit and credit
>
> [!WARNING]
> **🟡 Strong** — a mutable balance field (UPDATE accounts SET balance = balance - 100 WHERE id=?) fails regulatory requirements: you cannot reconstruct the history of how the balance arrived at its current value. Event sourcing: every financial movement is an immutable append-only event row: (account_id, event_type, amount, currency, timestamp, order_id, description). The current balance is computed as the sum of all events for an account
>
> [!TIP]
> **🟢 Staff+** — materializing the balance. Recomputing from the full event log on every balance check would be O(N) per query. Solution: maintain a checkpoint table (account_id, balance_as_of_timestamp) updated periodically (e.g., end of day). For live balance: checkpoint + sum of events since checkpoint = current balance. The checkpoint is always derivable from the event log — it's a cache, not the source of truth. If the checkpoint is wrong: recompute from event log (always possible, always correct). Regulatory requirement: event log must be immutable (no UPDATE or DELETE), retained for 7 years
#### Deep dive 3: Market data fan-out — high-frequency prices to millions of users
_Exchange feed delivers 10,000 price updates/second for thousands of tickers. Distributing this to 1M users watching various tickers is a fan-out problem_
> [!CAUTION]
> **🔴 Weak** — WebSocket to every user
>
> [!WARNING]
> **🟡 Strong** — topic-based pub/sub with ticker as the topic. Architecture: Market Data Service subscribes to exchange feed → normalizes updates → publishes to Kafka topics (one per ticker). App servers subscribe to tickers that their connected users are watching. On price update: app server pushes to relevant user WebSocket connections
>
> [!TIP]
> **🟢 Staff+** — app server maintains an in-memory map of (ticker → [user_connection_ids]). On price update event: look up connections for that ticker, push to each. This is a local fan-out within one server — no cross-server coordination needed because connections are affined by ticker. Hot tickers (GME, AAPL during earnings): the app server handling those users gets more messages but handles them without cross-server coordination. The Kafka topic for hot tickers may need multiple partitions to distribute ingestion load
_Why the deep dives connect to the scaling problem: "Real-time system plus financial workflow engine." Deep dive 1 solves order correctness. Deep dive 2 solves auditability. Deep dive 3 solves market data distribution._
Interview script
1. Financial-integrity script.
2. "Clarifying questions: are we building the full trading platform — equities, options, crypto — or just equities with market orders? And what are the latency requirements for order execution?"
3. "Good — equities, market and limit orders. Latency target: order submission confirmed to user within 500ms. Core features: view portfolio, get quotes, place orders, view order history. Out of scope: options, margin, tax optimization."
4. "Two non-negotiables for any financial system: idempotency and immutable audit trail. I'd establish both upfront."
5. "Idempotency: client generates UUID before every order request. Server stores idempotency_key as PENDING before calling broker. On retry: find key, query broker for status, update to COMPLETED or FAILED. Store before the call — not after — or a crash between them creates a duplicate order."
6. "Ledger: event sourcing. Every debit and credit is an immutable event row. Balance = sum(events) via a materialized checkpoint. Never UPDATE a balance field. Required for 7-year regulatory audit trail."
7. "Order execution: two-phase. (1) Reserve buying power in PG within an ACID transaction. (2) Call broker API with idempotency key. (3) On broker success: commit the debit and record the fill. On timeout: mark PENDING, reconcile async by re-querying broker."
8. "Market data pipeline: separate concern. Exchange feed → Kafka → Redis. App servers subscribe by ticker symbol. Fan-out to users watching that ticker via WebSocket. High-frequency, eventual consistency OK — 1-2 second staleness is fine for display."
Whiteboard
```
+-------------------+
| Mobile / Web |
| Clients |
+---------+---------+
|
HTTPS for API | SSE for live prices
|
+--------v--------+
| Load Balancer |
| sticky for SSE |
+---+----------+---+
| |
+--------------+ +----------------+
| |
+--------v--------+ +---------v---------+
| Order Service | | Symbol Service |
| create cancel | | SSE subscriptions |
| list orders | | fanout to clients |
+---+---------+---+ +----+----------+---+
| | | |
| | | |
| | subscribe by symbol |
| | | |
| | +-----v----------v-----+
| | | Redis Pub Sub |
| | | channels per symbol |
| | +-----------+----------+
| | |
| | |
| +-----v------------------+ |
| | Order DB | |
| | relational sharded by | |
| | userId | |
| +------------------------+ |
| |
| +------------------------+ |
+-->| ExternalOrderId KV |<-----------------+
| externalOrderId -> | trade lookup
| (orderId, userId) | |
+------------------------+ |
|
+--------v---------+
| Trade Processor |
| consumes exchange|
| trade feed |
+--------+---------+
|
updates price cache |
publishes symbol updates |
updates order state |
|
+-------------v--------------+
| Price Cache |
| latest price per symbol |
+-------------+--------------+
|
initial snapshot for SSE
|
|
outbound requests through small set of IPs
|
+-------v-------+
| NAT Gateway |
| / Egress GW |
+-------+-------+
|
sync place cancel APIs
|
async trade feed / webhook
|
+-------v---------+
| Exchange |
| order API + |
| trade feed |
+-----------------+
+-----------------------------+
| Cleanup Worker |
| scans pending and |
| pending_cancel orders |
| reconciles with exchange |
+-----------------------------+
```
The key idea is that you split the system into two flows. One flow is fast live price distribution through Trade Processor -> Redis -> Symbol Service -> SSE clients. The other flow is consistent order handling through Order Service -> DB first -> Exchange -> reconcile state.
If you draw this in an interview, you should call out three important choices. Use SSE for live prices, use a relational orders database partitioned by userId, and use a small egress layer so you do not open too many direct exchange connections.
Every edit is an operation (insert/delete at position + document version). The OT server serializes concurrent ops from multiple clients and transforms conflicting ops so all clients converge. Clients apply ops optimistically for low latency.
Problem
Real-time collaborative document editing where multiple users can type simultaneously and all see a consistent document.
The hard part: two users editing the same position concurrently — without OT, the document diverges.
Failures
**OT server is a single point of failure for an active document**
If the OT server crashes, all active editors lose their connection. In-flight operations are lost.
_Fix:_ OT server state is the op log, which is persisted to Cassandra/PG. On crash + reconnect, new OT server instance loads op log and reconstructs document state. Client buffered operations replay from last acknowledged version. Recovery < 10 seconds.
**Op log grows indefinitely for long-lived documents**
Loading a document that has 10 years of individual keystrokes takes forever. Document load time = time to replay entire op log.
_Fix:_ Periodic snapshot: every N operations (e.g., every 100 ops), write a full document snapshot. On load: fetch latest snapshot + only the ops since the snapshot. Bounded load time regardless of document age.
**Two editors on slow connections cause excessive conflicts**
Slow client sends ops based on stale document version. OT server must transform against many intervening ops. Transform logic gets complex and slow.
_Fix:_ Version vector on every op. OT server rejects ops too far behind (>100 ops stale) and asks client to resync. Client downloads current state and resumes. Better UX than silently corrupting the document.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 2B total docs, 100M DAU, avg 5 active docs per active user, avg 10 ops/minute when active |
| Read QPS | Presence/cursor updates: 100M × 5 docs × 1 cursor update/10s = 50M cursor updates/s across all docs — distributed across millions of OT server instances |
| Write QPS | 100M × 5 × 10 ops/60s ≈ 833K op/s — heavily partitioned by document (each doc is independent) |
| Storage | Op log: 833K ops/s × 50 bytes × 86400 ≈ 3.6 TB/day if all kept. Snapshots + recent ops: much smaller. Retain last 30 days ops + snapshot = feasible. |
| Cache math | Active document state in OT server memory: avg 10 KB × 500K concurrently active docs = 5 GB — fits in OT server fleet memory. |
| Verdict | Partitioned by document, not by user. Each document is an independent unit of work. Scaling is near-linear: more documents = more OT servers. Hot documents (1000 concurrent editors) need dedicated OT server instances. |
Design decisions
**OT vs. CRDT for conflict resolution**
→ OT for Google Docs-style rich text
CRDT: commutative operations, eventual consistency without a central server. OT: strong consistency, server serializes operations. For rich text with complex formatting (bold spans, nested lists, tables), CRDT data structures are very complex and memory-heavy. OT is simpler to reason about for this content type.
_Revisit when:_ CRDTs work well for plain text (used by many collaborative note apps). For rich document structure, OT or CRDT with a merge function per content type.
**Document affinity: always route same doc to same OT server**
→ Consistent hash routing by document_id to OT server cluster
OT server maintains in-memory document state (current content + pending ops). If different operations for the same document land on different servers, they can't be serialized locally. Single server per document is required.
_Revisit when:_ Primary-secondary per document for HA — operations go to primary, secondary has replicated state for failover.
**Persistence: write-through vs. async write to op log**
→ Write-through: op log written to Cassandra before ACK to client
Document edits must not be lost. ACKing before persistence means a server crash loses the op. At 833K ops/s, Cassandra handles the write load (it's append-only, partitioned by doc_id).
_Revisit when:_ Async write with in-memory buffer (WAL pattern) for lower latency if Cassandra write latency is too high.
Follow-up Q&A
**How do you handle 1,000 simultaneous editors on a viral document?**
OT server for that document is a hot spot. Assign a dedicated server instance (or small cluster with primary-secondary) for high-concurrent documents. Detect via Prometheus metric: concurrent_editors_per_doc > threshold. Auto-migrate document to dedicated instance. Other documents unaffected.
**How does undo work in a collaborative document?**
Undo is per-user: undo my last op, not the globally last op. OT keeps per-user op history. Undo generates an inverse op that's sent through the same OT pipeline — it gets transformed against all ops since the one being undone, then applied. Complex but correct.
**How do you handle offline editing (airplane mode)?**
Client buffers ops locally while offline. On reconnect: client sends all buffered ops with version numbers. Server replays them through OT against all ops that happened since disconnection. If conflict is unresolvable (e.g., document was deleted), server notifies client.
**How do you display cursor positions for other editors?**
Cursor positions are ephemeral — not persisted to op log. Sent via a separate low-latency ephemeral channel (WebSocket or WebRTC data channel). OT server broadcasts cursor updates to all active editors. Cursors use the same OT position transformation so they stay in the right place as others type.
**How do you implement suggestions / track changes mode?**
Suggestions are ops with a special metadata flag: proposed = true, author = user_id. They render with strikethrough/highlight styling but don't modify the canonical document state. Accepting a suggestion applies the op normally. Rejecting deletes the pending op. These are regular OT operations with additional metadata — no change to the core OT logic.
**OT vs CRDT — when would you switch?**
OT when you want server-authoritative rich text with simpler conflict semantics. CRDT (Yjs) when you need peer-to-peer or offline-first with higher per-character memory cost. State the tradeoff explicitly — interviewers expect it.
**How do you version documents for 'restore to Tuesday'?**
Snapshot + op log: find snapshot before target time, replay ops until timestamp. Never delete ops — snapshots are acceleration, log is source of truth.
**How do you scale comment threads without overloading OT server?**
Comments are overlay metadata keyed by (doc_id, anchor_position). Stored separately from text ops. OT server broadcasts comment events on same WebSocket channel but does not merge comments into document OT sequence.
Evolution
**v1 — MVP** — Single server, last-write-wins. Locks document while editing. Only one editor at a time. Breaks any collaborative use case.
**v2 — OT collaboration** — OT server per document. Op log in Cassandra. Snapshots every 100 ops. WebSocket delivery. Handles real-time collaboration for millions of documents.
**v3 — Enterprise scale** — Document affinity with consistent hashing. Hot document dedicated instances. Offline editing with op buffering. Comments as separate overlay system. Real-time presence and cursors.
Why it's hard to scale
The hard part in Google Docs is concurrent editing on the same shared document. You are not just storing text. You are merging many tiny edits from different users, keeping everyone’s screen nearly in sync, and making sure the document is still correct after races and reconnects.
There are three main scaling pain points. First, consistency is tricky because two users can edit the same spot at the same time, so naive last write wins will lose data or corrupt positions. Second, the system is real time and stateful. Each active document has connected editors, cursor positions, and a stream of low latency updates, which is much harder than normal stateless HTTP traffic. Third, the hot spot is per document. Most docs are quiet, but one shared doc can suddenly have many active editors all sending and receiving updates at once, so you need to route everyone for that doc to the right place and recover cleanly if that server fails.
A fourth issue is storage shape. If you store every keystroke forever, loading a document gets slower and storage keeps growing, so you usually compact old edits into snapshots. The short interview answer is this. Google Docs is hard because it combines concurrent write correctness, real time fan out, and per document hotspot state in one system.
Key points
- **Scope it first** — Core: real-time collaborative text editing with conflict resolution, persistent document storage, version history. Out of scope unless asked: comments, suggestions, offline mode, access control, spreadsheets.
- **OT server serializes all ops** — Every edit = (type, position, content, version). OT server transforms concurrent ops so all clients converge to the same document. transform(op_A, op_B) → op_A_prime that accounts for op_B having happened first.
- **Optimistic local application** — Client applies op immediately (low latency UX). When server sends the transformed op back, client reconciles. Users see their own keystrokes instantly — server confirms within ~50ms.
- **Op log + snapshots = bounded load time** — Every op is an immutable append to Cassandra. Snapshot every 100 ops. On document load: fetch latest snapshot + ops since snapshot. Load time is O(recent ops), not O(total history).
- **Document affinity — one OT server per doc** — All editors for a document must route to the same OT server (consistent hash by doc_id). OT cannot be distributed across servers for the same document without cross-server ordering — never shard one doc's ops.
- **CRDT as the alternative** — Commutative data structures — no central server needed. Better for offline-first. Higher per-character memory overhead. Yjs (YATA algorithm) is the production CRDT for rich text. OT is simpler for server-authoritative systems.
- **Failure mode to name** — OT server crashes mid-session: all clients disconnect. Warm standby (replicated op log from Cassandra) promotes in <10s. Clients reconnect, send buffered ops from last acknowledged version. Op log is always the source of truth.
> OT: requires central server. CRDT: no server needed but more memory. Both achieve convergence.
Tradeoffs
**OT vs CRDT** — OT requires a central server to serialize and transform ops — strong consistency. CRDT: commutative operations, no server needed, better for offline-first. OT is simpler for rich text; CRDT has higher per-character memory overhead.
**Snapshot + log vs log only** — Log-only is simpler but reconstructing a large document requires replaying thousands of ops — unbounded load time. Snapshot every N ops bounds reconstruction to snapshot + recent ops only, regardless of document age.
**Synchronous op log write vs async** — Write-through (ack after Cassandra write) guarantees no op is lost if server crashes. Async write (ack before Cassandra) is lower latency but can lose ops on crash. For a document editor, data loss is unacceptable — write-through.
**Single OT server per doc vs sharded** — Single OT server serializes all ops for a document — correct but single point of failure. Primary + warm standby with replicated op log gives HA. Sharding across multiple OT servers for one doc requires cross-shard ordering — avoid.
> "OT and CRDT both achieve convergence. OT requires a central server. CRDT doesn't but uses more memory. Google Docs uses OT."
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
#### Deep dive 1: Operational Transform — the concurrency correctness algorithm
> [!CAUTION]
> **🔴 Weak** — last-write-wins — the most recent save overwrites earlier concurrent edits. Data loss is guaranteed for any concurrent editing session
>
> [!WARNING]
> **🟡 Strong** — Operational Transform. Every edit is an operation with type, position, content, and the document version the client was on when they made the edit. The OT server serializes all operations: when concurrent ops arrive, it transforms each op against the ops that happened since the client's version
>
> [!TIP]
> **🟢 Staff+** — OT vs CRDT. OT requires a central server to serialize and transform — strong consistency, simpler for rich text. CRDT: commutative operations, no central server needed, better for offline-first. Yjs (YATA algorithm) is the production CRDT for rich text. OT is correct for server-authoritative systems; CRDT is correct for peer-to-peer. State this tradeoff explicitly — it's the question interviewers ask
#### Deep dive 2: Document state persistence — snapshot + op log for bounded load time
> [!CAUTION]
> **🔴 Weak** — store every op in Cassandra, replay all ops on document load. For a document with 1M keystrokes, load time = time to replay 1M ops — unbounded and growing forever
>
> [!WARNING]
> **🟡 Strong** — periodic snapshots. Every 100 ops: write a full document snapshot to PostgreSQL. On load: fetch latest snapshot + ops since the snapshot. Load time is O(recent ops), not O(total history)
>
> [!TIP]
> **🟢 Staff+** — implementation detail: the snapshot interval is a tunable config, not a code constant. Large documents (100-page reports) need more frequent snapshots; small documents can go longer. The snapshot is always derivable from the op log — it's a cache, never the source of truth. If a snapshot is corrupted, you can always reconstruct from the full op log
#### Deep dive 3: Scaling beyond one OT server — document affinity and hot document handling
> [!CAUTION]
> **🔴 Weak** — shard the OT server horizontally — split documents across multiple OT servers for throughput
>
> [!WARNING]
> **🟡 Strong** — document affinity is mandatory. All ops for a single document must go to one OT server — distributed sharding across multiple servers for the same document requires cross-shard op ordering, which is the same problem OT was designed to solve. Consistent hash by doc_id routes all editors to the same server
>
> [!TIP]
> **🟢 Staff+** — hot document handling: monitor concurrent_editors per document. When >50 concurrent editors: provision a dedicated OT server instance for that document, isolated from other docs. Primary + warm standby per hot document: standby replicates the op log from Cassandra and can promote in <10 seconds on primary failure. This gives hot document HA without sharding the OT logic
_Why the deep dives connect to the scaling problem: "Concurrent write correctness, real-time stateful system, per-document hot spot." Each deep dive addresses one layer._
Interview script
1. OT-first script.
2. "Clarifying questions: are we building a collaborative text editor — like Google Docs for plain documents — or also spreadsheets and presentations? And what's the scale — thousands of concurrent editors per document, or millions of documents with a few editors each?"
3. "Good — text documents, millions of documents with 1-10 concurrent editors typically, with occasional hot documents at 100+. Core features: real-time collaborative editing, persistent document storage, version history. Out of scope: comments, permissions, offline mode."
4. "The hard problem — and I'd lead with this: two users type simultaneously at the same position. Without coordination, their changes conflict and the document diverges on each client."
5. "Solution: Operational Transform. Every edit is an operation: type (insert or delete), position, content, and the document version the client was on when they made the edit."
6. "The OT server serializes all operations. When two concurrent ops arrive — say A inserts at position 5 and B deletes at position 3 — the server transforms A's op against B's to compute where A's insert should actually land given that B's delete happened first. It then broadcasts the transformed op to all clients."
7. "Clients apply their own ops immediately (optimistic local application) for low-latency UX. When the server sends the transformed op back, the client reconciles — typically within 50ms."
8. "Persistence: every op is an immutable append to Cassandra, keyed by (doc_id, version). Snapshot every 100 ops — on document load, fetch latest snapshot plus ops since. Load time is O(recent ops), not O(total history)."
9. "Document affinity: all editors for the same document must route to the same OT server. I'd use consistent hash by doc_id. For hot documents — a company all-hands doc — assign a dedicated OT server instance."
Whiteboard
```
+------------------+
| Client |
| Web / Mobile |
+---------+--------+
|
HTTP + WebSocket
|
+--------v--------+
| API Gateway |
+--------+--------+
|
+----------------+----------------+
| |
| POST /docs | WS /docs/{docId}
| |
+--------v--------+ +--------v-------------------+
| Document Meta | | Document Service |
| Service | | owns active doc sessions |
+--------+--------+ | runs OT transform |
| | tracks presence in memory |
| +----+-------------------+---+
| | |
| | |
+--------v--------+ | |
| Postgres | | |
| Document MetaDB | | |
| docId, title, | | |
| versionId | | |
+-----------------+ | |
| |
append ops | | broadcast edits
| | and cursors
+-------v--------+ |
| Document Ops |<---------+
| DB Cassandra |
| partition by |
| documentId |
+----------------+
In memory inside Document Service per active document
documentId
-> active websocket connections
-> latest loaded operations or materialized doc state
-> pending unacked edits
-> cursor positions
-> presence list
```
The core idea is simple. Document Meta Service creates documents and stores lightweight metadata. Document Service handles live collaboration, receives edit operations over WebSocket, applies Operational Transformation, writes durable ops to Cassandra, then pushes the transformed updates to every connected editor.
If you want the scaled version, add this around Document Service.
```
Clients
|
v
+------------------+
| Load Balancer |
+--------+---------+
|
v
+-----------------------------------------------+
| Document Service Cluster |
| |
| +-----------+ +-----------+ +-----------+ |
| | Doc Srv A | | Doc Srv B | | Doc Srv C | |
| +-----+-----+ +-----+-----+ +-----+-----+ |
| \\\\ | // |
| \\\\ | // |
| +---------- v -----------+ |
| | Consistent Hash Ring | |
| | docId -> owning server | |
| +-----------+------------+ |
+----------------------+------------------------+
|
v
+------+------+
| ZooKeeper |
| ring config |
+-------------+
Each docId maps to one owning Document Service.
All editors for the same document connect to the same server.
That keeps OT and fanout simple.
```
If you want, I can also give you a cleaner interview-ready version with just 6 boxes so it is easier to draw under time pressure.
A consistent hash ring routes each key to its responsible node. Adding or removing a node remaps only K/n keys (modulo hashing remaps all). Virtual nodes (150+ per physical) ensure even load distribution. Each node uses a DLL + HashMap for O(1) LRU eviction.
Problem
Building a distributed in-memory cache that supports GET/SET, handles node failure gracefully, evicts LRU items under memory pressure, and rebalances efficiently when nodes change.
Failures
**A cache node fails unexpectedly**
K/n keys are temporarily unavailable. Application falls through to primary DB — thundering herd if many keys expire simultaneously.
_Fix:_ Replication: each node has a primary and 1-2 replicas. On primary failure, replica promotes (Sentinel-managed election). Keys available from replica within ~30 seconds. DB protection: circuit breaker limits DB fallback throughput.
**Hot key: one cache key accessed 1M times/second**
Single node overwhelmed. Latency spikes for all keys on that node.
_Fix:_ Read replicas for hot keys (multiple copies). Local in-process L1 cache on each application server (100ms TTL, LRU eviction). Hot key detection: monitor access frequency per key, alert at > 10K QPS.
**Network partition: cache cluster splits into two halves**
Each partition continues serving reads from its subset of keys. Writes to either partition are invisible to the other. When partition heals, state diverges.
_Fix:_ During partition: serve reads from whichever partition the client reaches (availability over consistency). On partition heal: use last-write-wins (LWW) with vector clocks. Expired keys automatically resolve: TTL means stale data self-heals within the TTL window.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 10K application servers, 1M QPS total cache traffic, 100 GB data set, 1 KB avg value size |
| Read QPS | 1M QPS across 10 cache nodes = 100K QPS/node. Redis handles 500K ops/s/node — fine. |
| Write QPS | Write ratio 20% = 200K writes/s across 10 nodes = 20K writes/node — fine. |
| Storage | 100 GB data / 10 nodes = 10 GB/node. Modern servers: 64-128 GB RAM. Plenty of headroom. |
| Cache math | Consistent hash: adding an 11th node moves 100GB/11 = 9GB of data — 9% of keys remapped. vs. modulo hash: 100% of keys remapped. This is the fundamental argument for consistent hashing. |
| Verdict | Hot key handling and failure mode (thundering herd on node failure) are the real challenges. Consistent hashing + virtual nodes is the foundation but not the whole answer. |
Design decisions
**Consistent hashing vs. modulo hashing for key distribution**
→ Consistent hashing with 150 virtual nodes per physical node
Modulo hashing: adding node N+1 remaps N/(N+1) = ~100% of keys. During remapping, all cache misses = thundering herd on DB. Consistent hashing: adding one node remaps K/N keys (~10%). 150 vNodes per physical node gives even distribution (without vNodes, distribution variance can be 3×).
_Revisit when:_ Rendezvous hashing as a simpler alternative with the same O(K/N) remapping guarantee.
**LRU vs. LFU eviction policy**
→ LRU as default, LFU available for frequency-stable workloads
LRU: O(1) with DLL + HashMap. Works well for temporal locality (recently used likely to be used again). LFU: better for stable hot/cold split but requires frequency counter per key (memory overhead) and doesn't handle cold-start well (new popular key starts with count=0).
_Revisit when:_ Redis 4.0+ implements LFU with approximation. Use LFU when access patterns are stable and working set is predictable.
**Persistence: RDB snapshots vs. AOF log vs. none**
→ RDB snapshots for cache-as-cache use case, AOF for cache-as-primary
If cache is purely a cache (DB is source of truth): RDB is fine. On failure, reload from DB. If cache is used as primary storage: AOF (append-only log) gives durability at the cost of 2× write throughput.
_Revisit when:_ For a distributed cache system design interview, always clarify: is cache a cache or a durable store? Different answer for each.
Follow-up Q&A
**How do you handle cache stampede when a popular key expires?**
Three strategies: (1) Probabilistic early expiration: re-compute slightly before expiry to avoid thundering herd. (2) Mutex lock: first client to find expired key locks it, recomputes, others wait. (3) Stale-while-revalidate: serve stale value while async recompute runs. Option 3 is best for most use cases — users prefer slightly stale over waiting.
**What happens to in-flight writes during a node failure?**
With replication: write goes to primary, synchronously replicated to replica before ACK (strong consistency) OR async replicated (higher throughput, potential data loss on primary failure). The choice depends on durability requirement. For a cache, async replication is usually fine.
**How do you handle cache invalidation across multiple services?**
Tag-based invalidation: key tagged with entity_id. On entity update, invalidate all keys with that tag. Implementation: tag is a version counter in Redis. Key embeds version: user:123:v5. On update, increment version. Old keys become unreachable (and eventually evicted). No explicit invalidation broadcast needed.
**How would you implement a distributed lock on top of this cache?**
SETNX key value PX ttl_ms: atomic set-if-not-exists with TTL. Returns 1 (acquired) or 0 (already held). On success, caller holds lock for TTL duration. Release: verify key value (compare-and-delete via Lua script to avoid releasing someone else's lock). This is Redlock's basic building block.
**How do you do a zero-downtime node addition?**
(1) Add new node to consistent hash ring with virtual nodes. (2) New node starts receiving writes for its key range immediately. (3) Reads to new node: cache miss → fetch from DB → populate. No migration needed — cache re-warms naturally from read traffic. Keys on old nodes eventually evict. Zero disruption to application.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: end-to-end latency, consistency lag, fan-out depth. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: active-active or warm standby; conflict resolution on merge. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — Single node** — Single Redis instance. Modulo hash on client side for theoretical multi-key ops. Works until data > RAM or single node throughput limit.
**v2 — Distributed** — Consistent hashing with virtual nodes. Leader-follower replication per shard. Sentinel for automatic failover. Handles petabyte-scale datasets.
**v3 — Optimized** — Hot key detection and read replicas. LFU eviction for stable workloads. Tag-based invalidation. Client-side L1 cache for hottest keys. Redlock for distributed locking.
Why it's hard to scale
The hard part is that a distributed cache stops being just a fast in memory map and becomes a coordination problem across many machines.
There are three big scaling pain points. First, you need to shard data across nodes so each key lands on the right machine, and adding or removing nodes should not force you to reshuffle almost everything. That is why consistent hashing matters. Second, you usually want high availability, which means replicas, and replicas create sync problems because reads can become stale and failover gets tricky. Third, hot keys break the nice even distribution. One popular key can overload a single shard even if the rest of the cluster is idle.
A fourth issue is that network cost starts to matter. On one machine, a hash lookup is tiny. In a distributed cache, every get and set may involve a network hop, connection management, and sometimes cross node coordination. So the short interview answer is that distributed cache is hard to scale because you need to keep latency low while handling sharding, replication, rebalancing, and hot spots at the same time.
Key points
- **Scope it first** — Core: GET/SET with TTL, LRU eviction, consistent hashing across nodes, replication for HA. Out of scope unless asked: persistence to disk, Pub/Sub, Lua scripting, sorted sets.
- **Consistent hashing — non-negotiable** — Modulo hashing: adding one node remaps all K keys → simultaneous cache miss on every key → thundering herd → database crash. Consistent hashing remaps only K/N keys (~10%). This is not an optimization — it is required for safe topology changes.
- **Virtual nodes for even distribution** — Without vNodes, random ring positions give 3× variance in load across nodes. 150 vNodes per physical node: each node gets 150 ring segments. Adding a new node takes small slices from many existing nodes — balanced from day one.
- **LRU = DLL + HashMap, O(1) both** — Doubly-linked list for recency order (head = MRU, tail = LRU). HashMap for O(1) key → node lookup. get: HashMap lookup + move node to head. evict: remove tail node + delete from HashMap. Both O(1), no approximation needed.
- **Async replication for HA** — Primary + 1-2 replicas per shard. Async replication: lower write latency, potential loss of last few writes on primary failure. For a cache (data rebuildable from DB), this tradeoff is correct. Sync replication doubles write latency — wrong for cache.
- **Hot key handling** — One key accessed 1M/sec overloads one shard. Two layers: (1) local in-process L1 cache on app server (100ms TTL, LRU of top 100 keys — zero network), (2) read replicas for detected hot keys. Hot key detection: monitor per-key QPS, alert at >10K/sec.
- **Failure mode to name** — Node failure during topology change: consistent hashing means only K/N keys are affected. Those keys miss to the DB (thundering herd risk). Circuit breaker: limit DB fallback rate to DB's sustainable write rate. Replica promotes in <30s via Sentinel.
> Mental model: consistent hash for routing, virtual nodes for balance, DLL+HashMap for LRU, gossip for failure detection.
Tradeoffs
**Consistent hashing vs modulo hashing** — Consistent hashing: add/remove node remaps K/N keys (~10%). Modulo: remaps all K keys — causes a thundering herd on the database during any topology change. Consistent hashing is non-negotiable at scale.
**LRU vs LFU eviction** — LRU evicts least recently used — good for temporal locality (recently used items likely needed again). LFU evicts least frequently used — better for stable hot/cold splits but poor cold-start behavior. LRU is the correct default.
**Async replication vs sync replication** — Sync replication: zero data loss, higher write latency (wait for replica ACK). Async: lower latency, potential loss of last few writes on primary failure. For a cache (data rebuildable from DB), async replication is the right tradeoff.
**Write-through vs cache-aside vs write-behind** — Cache-aside (app manages cache): most flexible, handles miss gracefully. Write-through (write to cache and DB together): simpler consistency but couples DB and cache latency. Write-behind (write to cache, async flush to DB): fastest writes, risk of data loss. Cache-aside is the standard interview default.
> "Consistent hashing: O(K/n) key remapping on node change. Modulo = O(K) = catastrophic at scale. This is the core insight."
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
#### Deep dive 1: Consistent hashing — O(K/N) remapping vs. O(K) for modulo
> [!CAUTION]
> **🔴 Weak** — use modulo hashing — key maps to node hash(key) % N. Simple. Adding node N+1 remaps N/(N+1) ≈ 100% of keys simultaneously. Every key misses, every miss hits the database, the database crashes
>
> [!WARNING]
> **🟡 Strong** — consistent hashing — place each node at one or more points on a 0→2^32 ring. A key maps to the first node clockwise from its hash. Adding a node: it takes over the key range between itself and its predecessor. Only K/N keys (~10%) are remapped
>
> [!TIP]
> **🟢 Staff+** — without virtual nodes, random ring positions create 3× variance in load across physical nodes — one node gets 30% of keys, another gets 5%. 150 virtual nodes per physical node (each node occupies 150 ring positions) gives uniform distribution. Adding a new node takes small slices from many existing nodes simultaneously, keeping load balanced from day one
#### Deep dive 2: LRU eviction — O(1) get and O(1) evict with DLL + HashMap
> [!CAUTION]
> **🔴 Weak** — on every get, scan all entries to find the least recently used one to evict. O(N) eviction — unacceptable at any meaningful cache size
>
> [!WARNING]
> **🟡 Strong** — doubly-linked list + HashMap. HashMap stores (key → DLL node) for O(1) lookup. DLL maintains recency order: head = most recently used, tail = least recently used. get: HashMap lookup O(1), move node to head O(1). evict: remove tail O(1), delete from HashMap O(1). put: insert at head O(1), evict tail if at capacity O(1)
>
> [!TIP]
> **🟢 Staff+** — implementation: the DLL needs sentinel head and tail nodes to eliminate edge cases (empty list, single element). On get, moving a node from its current position requires unlinking from prev/next and relinking at head — all O(1) pointer operations. LFU as an alternative: tracks access frequency, evicts least frequently used. Better for stable hot/cold workloads but has O(log N) update cost and poor cold-start behavior (new popular key starts at frequency 1, immediately evictable)
#### Deep dive 3: Replication and failure handling — availability without sacrificing latency
> [!CAUTION]
> **🔴 Weak** — replicate synchronously — every write waits for replica ACK before confirming to the client. Correct, but doubles write latency
>
> [!WARNING]
> **🟡 Strong** — async replication for a cache. Cache data is derived — it can be rebuilt from the source of truth (the DB). Losing the last few milliseconds of writes on primary failure is acceptable; the data is just re-fetched on the next miss. Primary + 1-2 replicas per shard, async replication, Sentinel-managed automatic failover in <30 seconds
>
> [!TIP]
> **🟢 Staff+** — hot key handling: one key accessed 1M times/sec overloads one shard regardless of replication. Two layers: (1) local in-process L1 cache on each app server — top 100 keys, 100ms TTL, zero network hops; (2) read replicas for detected hot keys. Hot key detection: monitor per-key access frequency, alert at >10K QPS. The L1 cache is the most effective lever — it eliminates the hot key problem entirely for the highest-traffic keys
_Why the deep dives connect to the scaling problem: "Coordination across machines with low latency." Deep dive 1 solves distribution. Deep dive 2 solves eviction. Deep dive 3 solves availability and hot spots._
Interview script
1. Consistent-hashing-first script.
2. "Clarifying questions: is this a pure cache (data rebuildable from a source of truth) or a durable primary store? And what are our target operations — just GET/SET, or also sorted sets, pub/sub?"
3. "Good — pure cache, GET/SET with TTL. That shapes the consistency model: I can use async replication since data loss on failover is tolerable (we just re-fetch from DB)."
4. "The most important design decision — I'd start here: consistent hashing. Without it, adding or removing any node remaps all keys simultaneously. Every key misses. The database gets hit by the full load at once. That's a practical outage. Consistent hashing remaps only K/N keys."
5. "The ring: hash each node to a point on a 0–2^32 space. A key maps to the first node clockwise from its own hash. Adding a node: it takes over the keys between itself and its predecessor. 150 virtual nodes per physical node for even load distribution."
6. "LRU eviction: doubly-linked list for recency order plus a HashMap for O(1) lookup. get moves the node to the head. Evict removes from the tail. Both operations are O(1). No approximation needed."
7. "Replication: primary-replica per shard. Async replication — lower write latency, acceptable data loss since this is a cache. Sentinel manages automatic failover. Target RTO < 30 seconds."
8. "Hot key handling: I'd add local in-process L1 cache on each app server — top 100 keys, 100ms TTL. Eliminates network hops entirely for the hottest keys. Only cache misses reach Redis."
Whiteboard
```
+-------------------+
| Clients |
| app servers, APIs |
+---------+---------+
|
get set delete |
v
+---------------------------------+
| Cache Client Library / SDK |
| - consistent hash routing |
| - connection pooling |
| - write batching for hot writes |
| - hot key suffix logic |
+-----------+---------------------+
|
routes directly to owning shard node
|
-------------------------------------------------------------
| Distributed Cache Cluster |
| |
| Shard A Shard B Shard C |
| |
| +-----------+ +-----------+ +-----------+ |
| | Primary A |-------> | Replica A | | Replica A2| |
| | async repl| | read copy | | read copy | |
| +-----+-----+ +-----------+ +-----------+ |
| | |
| | in memory per node |
| v |
| +-------------------------------+ |
| | Hash map key -> node pointer | |
| | Doubly linked list for LRU | |
| | TTL expiry on entries | |
| | Background cleanup process | |
| +-------------------------------+ |
| |
| +-----------+ +-----------+ +-----------+ |
| | Primary B |-------> | Replica B |-------> | Replica B2| |
| +-----+-----+ +-----------+ +-----------+ |
| | |
| v |
| +-------------------------------+ |
| | Hash map + LRU list + TTL | |
| +-------------------------------+ |
| |
| +-----------+ +-----------+ +-----------+ |
| | Primary C |-------> | Replica C |-------> | Replica C2| |
| +-----+-----+ +-----------+ +-----------+ |
| | |
| v |
| +-------------------------------+ |
| | Hash map + LRU list + TTL | |
| +-------------------------------+ |
-------------------------------------------------------------
Hot read handling
hot:key
|
+--> hot:key#1 on Shard A
+--> hot:key#2 on Shard B
+--> hot:key#3 on Shard C
Reads pick one copy to spread load.
Writes update all copies asynchronously.
Hot write handling
counter:item42
|
+--> counter:item42:1 on Shard A
+--> counter:item42:2 on Shard B
+--> counter:item42:3 on Shard C
Writes are spread across suffixes.
Reads aggregate across shards.
```
The main mental model is this. Each cache node is just a fast in memory LRU cache, and the distributed part comes from sharding keys across many nodes with replication for availability. If you want, I can also give you a cleaner interview sized version that fits on one whiteboard.
Upload via TUS resumable protocol. Raw video lands in S3, triggering parallel transcode workers — each resolution processed simultaneously. Output: HLS segments in S3 behind CDN. ABR: client fetches HLS manifest, picks quality based on measured bandwidth.
Problem
Hosting and streaming video at YouTube scale — billions of videos, hundreds of millions of simultaneous viewers.
Two hard problems: ingestion and transcoding (raw video → multiple quality levels → CDN), and adaptive bitrate streaming.
Failures
**Transcode job fails for a specific codec or resolution**
Video uploaded but some quality levels unavailable. Users on slow connections see buffering or error.
_Fix:_ Idempotent transcode jobs with per-resolution retry. Track (video_id, resolution, status) in job table. Failed resolutions re-queued independently. Serve available resolutions while others are processing. Never block video availability on full transcode completion.
**CDN cache miss on first request after upload**
First viewer after upload triggers an origin S3 fetch. For viral videos, thousands of concurrent first-viewers all miss CDN.
_Fix:_ CDN pre-warming for verified/high-subscriber channels: push popular video segments to CDN edge nodes before publication. For cold content: origin shield (a secondary cache layer) reduces origin load from cache miss storms.
**Search index falls behind during a major news event**
Videos about breaking news don't appear in search for 10+ minutes.
_Fix:_ Dedicated fast-lane Kafka topic for recently uploaded videos. High-priority ES indexer with small batch size. Monitor indexing lag with alerting at >60s.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 2B users, 500M DAU, 500 hours of video uploaded/minute, 1B views/day |
| Read QPS | 1B views / 86400 ≈ 11,574 view QPS — but each view = multiple CDN segment requests. CDN absorbs >99% of bandwidth. |
| Write QPS | 500 hours/minute = 30,000 seconds of video/minute = 500 raw video files/min if avg 1hr each ≈ 8 uploads/s |
| Storage | 8 uploads/s × avg 10 GB raw × 86400 = 6.9 PB/day raw. After transcode + compression: ~20% = 1.4 PB/day. Over a year: 500 PB. |
| Cache math | Top 1% of videos (5M) get 95% of views. Cache these 5M × avg 2 GB = 10 PB at CDN edge. CDN hit rate target: >99%. Only 1% of bandwidth hits S3 origin. |
| Verdict | CDN cost is the dominant operating cost. Storage growth (500 PB/year) is the dominant infrastructure planning concern. Transcode compute is significant but one-time per video. |
Design decisions
**TUS resumable upload vs. simple multipart upload**
→ TUS protocol (resumable)
Videos are large (1-100 GB). Simple HTTP upload fails on any network interruption — user loses the whole upload. TUS: track chunks independently, resume from last successful chunk. Client-side chunking also enables parallel upload streams.
_Revisit when:_ S3 multipart upload achieves the same result natively. TUS provides a standardized protocol on top.
**HLS vs. DASH for adaptive bitrate**
→ HLS as primary (with DASH for non-Apple platforms)
HLS: Apple-native, required for iOS/Safari. DASH: more flexible, better for DRM, not natively supported in Safari. YouTube actually uses a proprietary format. In interviews: HLS is the safe default answer.
_Revisit when:_ MPEG-DASH if the product needs advanced DRM or non-Apple-centric platform support.
**Transcode: in-house vs. cloud encoding service**
→ Cloud-based transcode (AWS Elemental / GCP Transcoder API)
Building a transcode fleet at scale is a separate product. Elastic scaling for transcode bursts. Cost per minute is higher but operational cost is much lower.
_Revisit when:_ In-house transcode at YouTube's scale (where marginal compute cost savings justify the operational complexity).
Follow-up Q&A
**How do you handle 4K video playback for users with slow connections?**
ABR (Adaptive Bitrate Streaming): HLS manifest contains multiple quality variants. Player measures available bandwidth and switches to appropriate quality tier. Buffer management: maintain 30s buffer, switch up if buffer > 30s and bandwidth allows, switch down if buffer drops below 10s.
**How do you implement skip-ahead (scrubbing) efficiently?**
HLS segments are typically 6-10 seconds each. Scrubbing skips to the correct segment. For long videos, a thumbnail track (SSTV - spritesheet of thumbnails, one per 10s) is generated during transcode and served separately. Seeking downloads only the segment at the target timestamp, not everything in between.
**How do you handle copyright claims (Content ID)?**
Perceptual hash (pHash) of video fingerprint + audio fingerprint (AcrCloud-style). Compare against Content ID database at upload time. Match → flag for monetization routing or takedown depending on rights holder policy. This is a separate async pipeline, doesn't block video availability.
**How do you serve 1B views per day without S3 being overwhelmed?**
CDN is the answer. S3 origin receives at most 1% of requests (on CDN miss) = 115 QPS to origin. S3 handles this trivially. The expensive part is CDN egress bandwidth cost, not origin request count.
**How would you build offline viewing (download for offline)?**
DRM-protected offline: Widevine (Android) / FairPlay (iOS). User downloads encrypted HLS segments + license key (valid for 30 days). Offline player decrypts using cached key. Key expiry enforces rental window. License server is queried only once per download, not per view.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: end-to-end latency, consistency lag, fan-out depth. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: active-active or warm standby; conflict resolution on merge. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Single video format. FTP upload. Serve from S3 directly. No transcode. Works for a prototype.
**v2 — Scale** — TUS resumable upload. Async transcode pipeline (multiple resolutions). HLS with ABR. CDN for delivery. Elasticsearch for search. Handles millions of uploads.
**v3 — 2B users** — Multi-region deployment. Content ID for copyright. Live streaming as separate pipeline. Shorts/Reels format. Offline download with DRM. 500 PB/year storage.
Why it's hard to scale
The hard part in YouTube is not storing videos. It is handling huge video files on the upload path and serving smooth playback on the watch path.
There are three main scaling pain points. First, videos are large blobs, so uploads need multipart and resumable transfer straight to blob storage instead of passing through app servers. Second, playback is bandwidth sensitive. Users have different devices and network quality, so you usually split videos into small segments, transcode them into multiple qualities, and let the client switch between them during playback. Third, reads are extremely skewed. A video is uploaded once but may be watched millions of times, so popular videos create hot spots and you need CDN caching for segments and manifests plus caching for metadata.
The extra wrinkle is post processing. One upload turns into a pipeline that splits, transcodes, and writes many output files, which is a lot of CPU work even before anyone watches the video. So the short interview answer is this. YouTube is hard to scale because it combines large file uploads, expensive video processing, and massive read heavy streaming with hot viral traffic.
Key points
- **TUS resumable upload** — Chunked upload protocol. If upload fails, resume from last successful chunk.
- **Async parallel transcode** — S3 upload triggers a job. Separate workers for each resolution run in parallel. Output: HLS/DASH segments.
- **CDN = cost driver** — CDN absorbs 99%+ of bandwidth. App servers handle only metadata APIs.
- **ABR streaming** — HLS manifest lists quality levels. Client measures bandwidth and requests the appropriate tier.
- **Elasticsearch for search** — Full-text search over titles and descriptions.
> Transcode is parallelizable — all resolutions at once. CDN absorbs 99% of bandwidth. ABR = client picks quality tier.
Tradeoffs
**Parallel vs sequential transcode** — Parallel transcode per resolution is faster for users (360p available in seconds) and uses elastic compute efficiently. Sequential is cheaper but blocks all resolutions behind the slowest. Parallel always wins for UX.
**HLS vs DASH** — HLS is Apple-native, required for iOS/Safari, widely supported. DASH is more flexible for DRM and non-Apple platforms. HLS is the safe interview default; mention DASH if multi-platform DRM is required.
**CDN TTL vs purge on delete** — Long CDN TTL (24h+) maximizes hit rate and reduces origin cost. Deleted or copyright-struck videos must be purged instantly. Tag-based CDN purge (all segments of a video_id) reconciles both — long TTL by default, instant purge on demand.
**TUS resumable vs simple multipart upload** — Simple HTTP upload fails silently on any network drop — user loses the entire upload. TUS tracks chunks independently, resumes from last committed chunk. For files up to 100 GB on mobile connections, resumability is not optional.
> "Transcode parallelism, CDN as cost driver, and ABR for quality selection are the three concepts that define this system."
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
#### Deep dive 1: Upload pipeline — TUS resumable protocol and parallel transcode
> [!CAUTION]
> **🔴 Weak** — standard HTTP multipart upload, transcode sequentially through all resolutions. A 10 GB upload that fails at 9.9 GB restarts from zero. Sequential transcode means 360p isn't available until 4K finishes — user waits minutes before their video is watchable
>
> [!WARNING]
> **🟡 Strong** — TUS resumable protocol. Client splits the file into 10 MB chunks, tracks each independently. On network failure: resume from the last acknowledged chunk. Parallel transcode: one Flink job per resolution running simultaneously on separate worker instances. 360p is available within seconds of upload completion; 4K follows asynchronously
>
> [!TIP]
> **🟢 Staff+** — : transcode job fails for a specific resolution. Store transcode status per (video_id, resolution). Failed resolutions retry independently. Video is available at successful resolutions while others process. Never block video availability on full transcode completion — this is the difference between a 2-minute and a 20-minute time-to-publish
#### Deep dive 2: Adaptive bitrate streaming — HLS manifest and quality switching
> [!CAUTION]
> **🔴 Weak** — serve one video quality to all users — high quality for everyone, or low quality to save bandwidth
>
> [!WARNING]
> **🟡 Strong** — HLS adaptive bitrate streaming. The master manifest (.m3u8) lists all quality variants with bandwidth requirements. Client downloads the master manifest, picks initial quality based on current bandwidth estimate, then switches dynamically per segment
>
> [!TIP]
> **🟢 Staff+** — switching logic: switch up if measured bandwidth > 1.3× current bitrate AND buffer > 30 seconds; switch down if measured bandwidth < 0.8× current bitrate OR buffer drops below 10 seconds. This asymmetry (requires more headroom to switch up than to switch down) prevents oscillation — the player doesn't constantly flip between quality tiers on variable connections. Server-side: HLS segments stored in S3 with deterministic URL structure /{video_id}/{resolution}/{segment_number}.ts. The CDN pre-fetches upcoming segments during playback because the next segment URL is predictable
#### Deep dive 3: CDN architecture — cache warming, multi-CDN, and origin shielding
> [!CAUTION]
> **🔴 Weak** — put a CDN in front of S3 and set a long cache TTL
>
> [!WARNING]
> **🟡 Strong** — multi-layer CDN strategy. (1) Multi-CDN routing: two providers, DNS routes to the one with lower measured P95 latency for that region. Failover in seconds. (2) Origin shielding: instead of all CDN POPs fetching independently from S3 on a miss, a small set of shield POPs (10-20 globally) fetch from S3 and local POPs fetch from the shield. Reduces S3 request rate from O(POPs × misses) to O(shields × misses)
>
> [!TIP]
> **🟢 Staff+** — cache warming: for large channels, the CDN push API pre-populates segments at relevant POPs before the video goes live. No cold cache for the first million viewers of a major release. Purge strategy: deleted or copyright-struck videos need instant CDN purge across all POPs. Tag-based purge: all segments of a video_id are tagged at upload time, purged with a single API call on deletion
_Why the deep dives connect to the scaling problem: "Large blob uploads, expensive transcoding, massive read-heavy streaming." Each deep dive addresses one layer._
Interview script
1. Upload then stream framing.
2. "I'll cover two flows: video upload + transcoding, and video playback with ABR."
3. "Upload: creator uses TUS resumable protocol — chunks the file, tracks progress, resumes if interrupted. Raw video lands in S3."
4. "Transcoding: S3 upload triggers a processing job. Separate worker instances process each resolution in parallel — 360p, 720p, 1080p, 4K simultaneously. Output: HLS segments and manifest in S3 behind CDN."
5. "Playback: viewer requests a video. Video API returns metadata and a CDN URL for the HLS manifest. Client fetches the manifest, measures available bandwidth, and requests the appropriate quality. ABR switches quality as bandwidth changes."
6. "CDN is the critical infrastructure here — absorbs 99%+ of all bandwidth. App servers handle only API calls."
Whiteboard
```
+-------------------+
| Users |
| uploader, viewer |
+---------+---------+
|
HTTPS |
v
+-------------------+
| Load Balancer |
+---------+---------+
|
v
+-------------------+
| Video Service |
| stateless API |
+----+---------+----+
| |
| |
| +----------------------+
| |
v v
+------------------------+ +---------------------+
| Metadata Cache | | Video Metadata DB |
| distributed cache | | Cassandra |
+------------------------+ +---------------------+
UPLOAD FLOW
===========
1. Client asks for upload session and presigned URL
User ---> Video Service ---> Metadata DB
|
+--> returns videoId, multipart upload info,
presigned URLs
2. Client uploads video directly to blob storage
User ---------------------------------------------> S3 Blob Storage
multipart upload of raw video chunks
3. Client reports chunk progress
User ---> Video Service ---> Metadata DB
chunk uploaded status, ETag info
4. Upload completion triggers processing
S3 ObjectCreated event ---> Processing Orchestrator
PROCESSING PIPELINE
===================
+-------------------------+
| Processing Orchestrator |
| DAG workflow manager |
+-----------+-------------+
|
---------------------------------------------------
| | |
v v v
+----------------+ +----------------+ +----------------+
| Segment Worker | | Audio Worker | | Transcript |
| split raw file | | audio process | | Worker |
+-------+--------+ +--------+-------+ +--------+-------+
| | |
v v v
+----------------+ +----------------+ +----------------+
| Transcode | | audio outputs | | transcript out |
| Workers | | in S3 | | in S3 |
| many in parallel| +----------------+ +----------------+
+-------+--------+
|
v
+------------------------+
| Manifest Generator |
| primary + media files |
+-----------+------------+
|
v
+------------------------+
| S3 processed assets |
| segments, manifests |
+-----------+------------+
|
v
+------------------------+
| Metadata DB update |
| manifest URL, status |
| upload complete |
+------------------------+
PLAYBACK FLOW
=============
User ---> Video Service ---> Cache ---> Metadata DB
|
+--> returns manifest URL and metadata
User ---> CDN ---> S3 processed assets
| manifests and video segments
|
+--> edge serves cached content when possible
Client playback logic
- fetch manifest
- pick bitrate based on network
- download first segment
- keep downloading next segments
- switch quality up or down as bandwidth changes
FULL SYSTEM VIEW
================
+-------------------+
| Users |
+---------+---------+
|
v
+-------------------+
| Load Balancer |
+---------+---------+
|
v
+-------------------+
| Video Service |
+---+-----------+---+
| |
v v
+-------------+ +------------------+
| Cache | | Metadata DB |
| popular md | | Cassandra |
+-------------+ +------------------+
upload session / metadata | ^
| |
v |
+-------------------------------+
| S3 Blob Storage |
| raw uploads |
| processed segments/manifests |
+---------------+---------------+
|
object event |
v
+-------------------------------+
| Processing Orchestrator |
| workflow / DAG manager |
+---------------+---------------+
|
parallel worker fleet
|
-------------------------------------------------
| | |
v v v
+-------------+ +---------------+ +---------------+
| Split | | Transcode | | Other media |
| workers | | workers | | workers |
+-------------+ +---------------+ +---------------+
|
v
+----------------+
| Manifest Gen |
+--------+-------+
|
v
+-----------+
| CDN |
| edge cache|
+-----+-----+
|
v
Users
```
The mental model is two big paths. Upload goes client to S3, then processing pipeline, then metadata update. Watch goes client to metadata, then manifest, then CDN segment fetches.
If you want, I can also give you a smaller interview friendly version that fits in 60 seconds on a whiteboard.
The URL Frontier is a priority queue with per-domain rate limiting. Crawler workers fetch pages, extract links, respect robots.txt. New links → Dedup Service: Bloom filter for fast probabilistic check, Cassandra for confirmation. Raw HTML → S3.
Problem
Systematically discovering and fetching web pages. Handle billions of URLs, respect site rate limits, avoid re-crawling duplicates, and recover from failures.
Hard parts: politeness, deduplication at scale, and prioritizing fresh content.
Failures
**Crawler enters a trap (infinite URL space generated dynamically)**
Crawler spends all capacity on one domain. Rest of the web is starved.
_Fix:_ Per-domain URL count limit (e.g., max 1M URLs per domain). Path depth limit (max 10 levels deep). Detect trap patterns: if URLs follow a counter pattern (page?id=1, page?id=2 ... page?id=1000000) — truncate.
**Bloom filter for URL dedup fills up (too many URLs)**
False positive rate rises above acceptable threshold. Legitimate new URLs are incorrectly identified as 'already visited'.
_Fix:_ Monitor Bloom filter load factor. When > 70% capacity: rotate to a new Bloom filter, move old one to cold storage. Use Counting Bloom Filter to support deletions (crawl frequency requires periodic re-crawl). Or: scalable Bloom filter that grows dynamically.
**DNS resolution becomes a bottleneck at high crawl throughput**
At 10K fetches/s, DNS lookups at 100ms each would cap throughput at 10K DNS QPS. DNS servers get hammered.
_Fix:_ Local DNS cache per crawler node (TTL-respecting). Pre-fetch DNS for domains in the near-term crawl queue. DNS cache miss ratio target < 5%.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 5B pages to crawl, re-crawl every 30 days, 100ms avg fetch time (including DNS + TCP + transfer) |
| Read QPS | 5B / (30 × 86400) ≈ 1,929 crawl fetches/s needed. At 100ms avg: 193 concurrent crawlers. |
| Write QPS | 1,929 pages/s → extract ~50 links/page = 96K new URL candidates/s → Bloom filter checks |
| Storage | 5B URLs × 200 bytes (URL + metadata) = 1 TB URL frontier store. Bloom filter for 5B URLs: at 1% false positive rate ≈ 9.6 GB. Raw page content: 5B × avg 100KB = 500 TB. |
| Cache math | Bloom filter: 9.6 GB fits in RAM per node — no DB needed for URL dedup fast path. Cassandra only for confirmation of uncertain positives. |
| Verdict | 96K URL checks/s against the Bloom filter is the high-frequency operation. DNS caching and politeness throttling limit effective throughput more than raw compute. |
Design decisions
**BFS vs. priority-based crawl order**
→ Priority queue based on PageRank estimate + freshness + domain authority
Pure BFS treats all pages equally. In practice, CNN.com's homepage is more valuable than a personal blog's page 47. Priority queue maximizes value of crawled content within resource constraints.
_Revisit when:_ BFS is fine for small-scale crawls. Priority becomes critical when crawl budget is limited.
**Centralized URL frontier vs. distributed frontier**
→ Centralized with consistent hash assignment of domains to crawler nodes
Centralized frontier provides global URL dedup and politeness enforcement. Consistent hash: each domain is assigned to one crawler node, which enforces per-domain rate limiting without coordination.
_Revisit when:_ Distributed frontier (each crawler maintains a local queue) reduces coordination overhead but complicates global dedup.
**Respect robots.txt vs. ignore**
→ Always respect robots.txt — no discussion
Legal requirement (most jurisdictions). Practical requirement (violating robots.txt leads to IP bans). Cache robots.txt per domain (TTL = 24h). Check before every fetch.
_Revisit when:_ Never negotiate on this.
Follow-up Q&A
**How do you handle JavaScript-rendered pages (SPAs)?**
Headless browser (Puppeteer/Playwright) for JS rendering. Expensive: ~10× slower than simple HTTP fetch, ~100 MB RAM per instance. Use selectively: only for high-value domains known to require JS rendering. Most crawled content is static HTML.
**How do you detect and handle duplicate content (same content, different URLs)?**
SimHash (Locality Sensitive Hash) of page content. Hamming distance < 3 = near-duplicate. Canonical URL from HTML . Store canonical URL in URL frontier. Near-duplicate detected after fetch — content is stored but marked as duplicate in the index.
**How do you freshness-schedule re-crawls?**
Change frequency estimation: track how often a page changes (ratio of content changes per re-crawl). High-change pages (news sites): re-crawl every hour. Low-change pages (static content): re-crawl every 30 days. Schedule based on estimated next-change time, not fixed interval.
**How do you handle login-required content?**
Generally: don't crawl login-gated content (no authorization). Exception: explicit partnership with site (authenticated crawling with OAuth credentials). Social network crawlers maintain long-lived sessions with their own accounts.
**How do you distribute 1,929 fetches/s across crawlers fairly?**
Consistent hash assigns domain → crawler node. Each node gets a share of domains. Within a node: token bucket per domain (respects crawl-delay from robots.txt). Global load balancing: if one node falls behind, domain reassignment via consistent hash ring adjustment.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: end-to-end latency, consistency lag, fan-out depth. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: active-active or warm standby; conflict resolution on merge. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Single-threaded BFS crawler. Python + requests. SQLite for visited URLs. Crawls ~1 page/second. Good for a search engine prototype.
**v2 — Distributed** — Consistent hash URL frontier. Bloom filter dedup. Per-domain politeness queues. Kafka for URL distribution. Cassandra for visited URLs. Handles 1B pages.
**v3 — Production quality** — JS rendering with headless browser pool. SimHash content dedup. Freshness-based re-crawl scheduling. robots.txt cache. Trap detection. Multi-datacenter deployment.
Why it's hard to scale
The hard part in Web Crawler is not storing pages. It is coordinating a huge number of fetches across the public internet without wasting work or being rude to other sites.
There are four scaling pain points you should call out. First, the crawl frontier gets huge. You keep discovering new URLs, but you need to dedupe them so different workers do not crawl the same page again and again. Second, politeness limits parallelism. You may want massive throughput, but you still need per domain rate limits and robots.txt checks, so scaling is not just adding more workers. Third, the internet is unreliable. DNS lookups, slow servers, dead links, retries, and crawler traps all waste time unless you pipeline the work and track progress carefully. Fourth, the workload is very uneven. Some domains are tiny, some are enormous, and some generate endless near-duplicate pages, so load balancing is messy.
A good interview summary is this. Web Crawler is hard because it combines massive frontier management, external bottlenecks like DNS and website limits, duplicate avoidance, and fault tolerance in one system.
Key points
- **Scope it first** — Core: crawl the web, extract text, store pages, enable search indexing. Out of scope unless asked: JavaScript rendering, login-gated content, real-time re-crawl, entity extraction.
- **URL frontier is the core data structure** — Two-tier priority queue. Back queues: one per domain (enforces politeness — one request per domain per N seconds). Front queue: priority-ordered list of domains to crawl next. Never crawl faster than the site allows.
- **Always respect robots.txt** — Legal requirement in most jurisdictions. Cache per domain (TTL 24h). Check before every fetch. Violation = IP ban. State this proactively in interviews — it signals production awareness.
- **Bloom filter for URL dedup** — 96K new URL candidates/sec. DB-only dedup = impossible. Bloom filter (9.6 GB for 5B URLs at 1% FPR): O(1) per check. False positive = occasionally skip a valid URL — acceptable. Cassandra exact check only for Bloom positives.
- **DNS caching per domain** — At 1,929 fetches/sec, uncached DNS at 100ms per lookup = DNS becomes the bottleneck. Cache per domain with TTL (1 hour). Pre-fetch DNS for domains in the near-term frontier queue. DNS miss rate target < 5%.
- **Content dedup — SimHash** — 30-40% of the web is near-duplicate content. SimHash fingerprint: 64-bit, Hamming distance < 3 = near-duplicate. Partition fingerprints by first K bits for efficient lookup (LSH). Canonical URL from takes priority.
- **Failure mode to name** — Crawler enters a trap (infinite URL space): per-domain URL count limit (1M max), path depth limit (10 levels), counter-pattern detection. Without traps, one rogue domain can starve the entire crawl budget.
> Mental model: URL frontier for politeness + priority. Bloom filter + Cassandra for dedup. S3 for raw content.
Tradeoffs
**BFS vs priority-queue traversal** — BFS discovers breadth-first — finds popular pages sooner but treats all pages equally. Priority queue (weighted by PageRank estimate + domain authority) maximizes crawl value per unit of resource. Priority queue is the production choice.
**Bloom filter vs DB-only for dedup** — DB-only check at 96K URL candidates/sec would require 96K queries/sec — impossible. Bloom filter (9.6 GB for 5B URLs at 1% false positive) handles this in memory at O(1) per check. False positives just mean occasionally skipping a valid URL.
**Centralized frontier vs distributed** — Centralized frontier provides global dedup and politeness enforcement — simpler to reason about. Distributed frontier reduces coordination overhead at extreme scale but complicates dedup and rate limiting. Centralized is correct up to billions of URLs.
**Respect robots.txt vs ignore** — This is not a tradeoff — always respect robots.txt. Legal requirement in most jurisdictions. Practical requirement: violating it leads to IP bans. Cache robots.txt per domain (TTL 24h). State this proactively; it signals production awareness.
> "Bloom filter = fast probabilistic check before the expensive DB lookup. Per-domain rate limiting = politeness. robots.txt = required compliance."
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
#### Deep dive 1: URL frontier — priority queue, politeness, and domain assignment
> [!CAUTION]
> **🔴 Weak** — a single queue of URLs to crawl in BFS order — simple and correct for small scale
>
> [!WARNING]
> **🟡 Strong** — two-tier priority frontier. Back queues: one per domain, enforces politeness (one request per domain per N seconds, respects robots.txt Crawl-delay). Front queue: priority-ordered list of (domain, priority) pairs. The scheduler picks the highest-priority domain whose next_allowed_crawl_time ≤ now. This ensures politeness and value-maximization simultaneously
>
> [!TIP]
> **🟢 Staff+** — priority = f(PageRank estimate, domain authority, freshness score). High-authority domains (CNN, Wikipedia) get crawled most frequently. Pages within a domain are prioritized by inbound link count. The priority function is configurable — it's the lever that determines what percentage of your crawl budget goes to fresh high-value content vs. long-tail pages
#### Deep dive 2: URL deduplication — Bloom filter + Cassandra two-tier
> [!CAUTION]
> **🔴 Weak** — store all crawled URLs in a database, check before each fetch. At 96K new URL candidates/sec, 96K DB queries/sec for dedup alone — impossible
>
> [!WARNING]
> **🟡 Strong** — Bloom filter as a fast first-pass. At 5B URLs, 1% false positive rate: 9.6 GB memory, 7 hash functions, O(1) per check. False positive = occasionally skipping a valid URL — acceptable tradeoff. Cassandra exact check only for Bloom filter positives (~1% of checks)
>
> [!TIP]
> **🟢 Staff+** — operational concern: Bloom filter fills up. Monitor load factor — alert at 70% capacity, rotate to a new filter. During rotation: new URLs go to the new filter; old filter kept read-only for 30 days to catch duplicates of recently-crawled pages. Counting Bloom filter (supports deletions) if you need to remove URLs that have been re-crawled and should be treated as fresh
#### Deep dive 3: Content deduplication and relevance scoring — SimHash and PageRank
> [!CAUTION]
> **🔴 Weak** — detect exact duplicates using SHA-256 of page content. Misses near-duplicates — the same article republished on 50 domains with minor wording differences all get indexed, wasting storage and diluting search quality
>
> [!WARNING]
> **🟡 Strong** — SimHash fingerprint. Process: tokenize → tf-idf weighted term vector → hash each term with a random 64-bit weight vector → sum → take sign of each bit. Result: 64-bit fingerprint where similar documents have high Hamming similarity. Hamming distance < 3 = near-duplicate
>
> [!TIP]
> **🟢 Staff+** — lookup efficiency: comparing a new fingerprint against all 5B stored fingerprints is O(N). Solution: partition fingerprints into bands (groups of bits) and only compare fingerprints in the same band — Locality Sensitive Hashing. Reduces lookup from O(N) to O(1) average case. For 5B fingerprints × 8 bytes = 40 GB total — fits in a distributed Redis cluster for fast lookups
_Why the deep dives connect to the scaling problem: "Massive frontier, external bottlenecks, duplicate avoidance, fault tolerance." Each deep dive addresses one constraint._
Interview script
1. Frontier-first script.
2. "Clarifying questions: are we building a general-purpose crawler for a search engine, or a focused crawler for a specific domain? And what are the freshness requirements — how often should pages be re-crawled?"
3. "Good — general-purpose, re-crawl high-value pages weekly, low-value monthly. Core features: discover and fetch pages, extract text and links, detect duplicates, feed a search index. Out of scope: JavaScript rendering (unless asked), login-gated content."
4. "The URL frontier is the central data structure — I'd design it first. Two-tier queue: back queues grouped by domain (enforces politeness, one request per domain per N seconds), front queue that priority-selects which domain to crawl next (PageRank + freshness score)."
5. "Always respect robots.txt — legal requirement, and violating it leads to IP bans. Cache per domain with 24-hour TTL. Check before every fetch. State this proactively — it signals production awareness."
6. "URL dedup: Bloom filter first. At 96K new URL candidates/sec, DB-only check is impossible. Bloom filter (9.6 GB for 5B URLs, 1% FPR) handles this in memory at O(1). Cassandra exact-check for Bloom positives only."
7. "Content dedup: SimHash fingerprint per page. Hamming distance < 3 = near-duplicate — skip indexing. Catches the same article reposted on 50 domains. Canonical URL from takes priority."
8. "DNS caching: at 1,929 fetches/sec, uncached DNS at 100ms per lookup makes DNS the bottleneck. Cache per domain with TTL. Pre-fetch DNS for domains in the near-term frontier queue."
Whiteboard
```
+-------------------+
| Seed URL Input |
+---------+---------+
|
v
+-------------------------------+
| URL Frontier Queue SQS |
+-------------------------------+
|
dequeue URL msg
|
v
+---------------------------------------------------+
| URL Fetcher Workers |
| - check URL dedup in Metadata DB |
| - read domain robots rules |
| - acquire per-domain lock in Redis |
| - enforce crawl delay and rate limit |
| - resolve DNS and fetch page |
+-------------------+-------------------------------+
| |
disallowed or delayed | fetched HTML
| v
| +------------------------+
| | Raw HTML Blob Storage |
| | S3 |
| +-----------+------------+
| |
| v
| +-------------------------------+
| | Processing Queue |
| | URL id or blob pointer |
| +---------------+---------------+
| |
| v
| +-------------------------------+
| | Text and URL Extraction |
| | Workers |
| | - parse HTML |
| | - extract text |
| | - extract outgoing links |
| | - hash content for dedup |
| +----------+----------+---------+
| | |
| | |
| | v
| | +-------------------+
| | | New URL Discovery |
| | +---------+---------+
| | |
| | check seen URL
| | check depth limit
| | |
| | v
| | +--------------------+
| | | URL Frontier Queue |
| | +--------------------+
| |
| v
| +-------------------------------+
| | Text Blob Storage S3 |
| +-------------------------------+
|
v
+-------------------------------+
| Retry with Backoff |
| SQS visibility timeout + DLQ |
+-------------------------------+
+--------------------+ +----------------------+ +------------------+
| Metadata DB | | Redis | | DNS Cache |
| - URL state | | - per-domain lock | | - domain to IP |
| - crawl depth | | - rate limiting | +------------------+
| - html pointer | +----------------------+
| - text pointer |
| - content hash |
| - robots rules |
| - last crawl time |
+--------------------+
Outside system
+------------------+ +------------------+
| DNS Providers | | External Websites|
+------------------+ +------------------+
```
If you are presenting this in an interview, the simplest way to walk through it is this. URLs enter the frontier queue, fetchers crawl pages politely, raw HTML goes to blob storage, parser workers extract text and links, text goes to storage, and new links go back into the frontier after dedup checks.
The two things that make this feel complete are the control plane pieces. Metadata DB tracks crawl state and dedup, while Redis handles per-domain coordination so you do not overload a site.
Click events → Kafka. Real-time path: Flink tumbling windows aggregate per ad_id, with a 1-minute watermark. Results land in ClickHouse. Batch path: Spark computes exact daily counts and reconciles. Click dedup: UUID + Bloom filter at ingestion.
Problem
Counting ad clicks accurately at massive scale for billing advertisers.
The core challenge: exact counting at this scale is expensive. The system must balance real-time approximation vs exact billing accuracy.
Failures
**Flink job crashes mid-window**
Partial window aggregates lost. When job restarts, window starts fresh. Some clicks are counted twice (from Kafka replay) or not at all (lost in-memory state).
_Fix:_ Flink checkpointing to S3/HDFS every 30 seconds. On restart, restore from checkpoint. Kafka retention covers the replay window. Exactly-once semantics via Flink's transaction sink (idempotent writes to ClickHouse).
**Hot ad partition: one viral ad gets 100× normal click volume**
Single Kafka partition handling all clicks for the hot ad is overloaded. Flink consumer for that partition falls behind. Click data delayed.
_Fix:_ Repartition hot keys: detect high-volume ad_ids, assign multiple Kafka partitions to them (key-based routing with partition count per ad_id). Flink state is partitioned by ad_id anyway — just needs more partitions to distribute load.
**Duplicate clicks from user double-tapping or network retry**
Advertiser billed for clicks that never happened. Financial integrity issue.
_Fix:_ Click dedup: each click event has a UUID generated client-side. Bloom filter at ingestion service checks UUID. Duplicate UUID → drop event. Bloom filter false positive rate < 0.01%. For billing-grade accuracy: exact dedup in DB for top-spend advertisers.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 10B ads served/day, 1% CTR = 100M clicks/day, 10M active ad campaigns |
| Read QPS | 100M clicks / 86400 ≈ 1,157 click events/s ingested |
| Write QPS | Flink aggregation: 1,157 events/s → ~100 ClickHouse INSERT/s (batch aggregated) |
| Storage | Raw events: 1,157/s × 200 bytes × 86400 × 30 days ≈ 600 GB/month raw event log |
| Cache math | Advertiser dashboard reads: 10M campaigns × 1 dashboard refresh/minute = 167K QPS. Must be served from ClickHouse + Redis cache (not recomputed from raw events). |
| Verdict | The 167K dashboard QPS vs 1,157 event QPS ratio shows reads are the harder scaling problem, not writes. ClickHouse pre-aggregation + Redis caching for the dashboard layer is essential. |
Design decisions
**Lambda (stream + batch) vs. Kappa (stream only with replay)**
→ Lambda Architecture
Advertising revenue requires exact billing. Stream layer: fast, approximate, good for real-time dashboards. Batch layer: exact, reconciled daily, used for invoices. Both are needed because advertisers dispute charges based on exact counts. Kappa (stream replay for corrections) has higher latency for corrections.
_Revisit when:_ Kappa if reconciliation latency of daily batch is acceptable for advertiser billing.
**ClickHouse vs. Cassandra for aggregated storage**
→ ClickHouse (columnar OLAP)
Dashboard queries: SELECT sum(clicks), sum(impressions) GROUP BY campaign_id, date WHERE campaign_id = 123 AND date BETWEEN ... — this is OLAP, not OLTP. ClickHouse columnar compression gives 10× better scan performance than Cassandra for these queries.
_Revisit when:_ Cassandra for time-series click storage (high write throughput). ClickHouse for aggregated analytics. Both can coexist.
**Click dedup: Bloom filter vs. exact set vs. Redis SET**
→ Bloom filter for fast path, exact DB check for billing-critical top advertisers
1,157 events/s × 1M clicks/event_window = 1B unique IDs to check. Bloom filter handles this in ~1 GB RAM with 0.01% false positive rate. For top 1K advertisers by spend: exact UUID check in Redis (smaller set, exact required for billing disputes).
_Revisit when:_ HyperLogLog for approximate distinct click count — different problem (count unique users, not dedup specific clicks).
Follow-up Q&A
**How do you handle click fraud?**
Layered approach: (1) dedup within session (same user, same ad, < 1s = duplicate), (2) IP rate limiting (> 100 clicks/hour from one IP), (3) ML model (click pattern analysis — bot traffic has different timing/behavior distributions), (4) manual review queue for high-value campaigns.
**How do you handle attribution (click → conversion)?**
Attribution is a separate pipeline: join click events with conversion events (purchase, signup) by user_id within attribution window (7-30 days). Multi-touch attribution: different weight for first click, last click, or linear. This requires long-term event correlation, not just click aggregation.
**How do you serve real-time dashboard updates to 10M advertisers?**
ClickHouse pre-aggregated tables: hourly rollups materialized by (campaign_id, hour). Dashboard query hits materialized table, not raw events. Redis caches dashboard snapshots with 1-min TTL. Flink streams recent (last-hour) data to Redis directly for sub-minute freshness.
**What's your SLA for click data appearing in dashboards?**
Real-time dashboard: P99 < 2 minutes. Billing-grade exact counts: end of day + 2 hours (batch job window). Make this explicit: advertisers see approximate real-time data and exact daily billing data — two different SLAs, two different pipelines.
**How would you add impression counting to this system?**
Impressions are 10× more frequent than clicks (100B/day). Same Kafka → Flink → ClickHouse pipeline but with higher throughput. Key difference: impression events are larger volume, less fraud-sensitive (nobody pays for impressions in CPC model). Separate Kafka topics, separate Flink jobs, same ClickHouse database with separate tables.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: end-to-end latency, consistency lag, fan-out depth. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: active-active or warm standby; conflict resolution on merge. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — Batch only** — Hourly MapReduce job over raw click logs. 1-hour staleness. Good enough for daily reporting but useless for real-time campaign optimization.
**v2 — Lambda Architecture** — Flink stream processing for real-time aggregates. Spark batch for daily exact counts. ClickHouse for serving. Bloom filter dedup. Handles 100M clicks/day.
**v3 — Scale + fraud** — ML click fraud detection. Attribution modeling. Budget pacing (stop serving when campaign budget exhausted, detected in near-real-time). 10B impressions/day.
Why it's hard to scale
The hard part in Ad Click Aggregator is that it looks like a simple counter system, but it is really a high write analytics pipeline with correctness requirements.
There are three main scaling pain points. First, the write path is heavy. Every click is an event, so you cannot do raw database writes and then run GROUP BY queries on demand. You need to buffer and pre aggregate the data. Second, freshness matters. Advertisers want near real time metrics, so pure batch processing makes data too stale, which pushes you toward streaming aggregation. Third, correctness matters a lot because clicks map to money. You cannot lose events, and you also do not want to double count duplicate clicks.
A fourth issue to call out is skew. Most ads are quiet, but one viral ad can create a hot shard if all clicks for that ad land on the same partition. That means the system is hard not because 10k clicks per second is huge by itself, but because you need high write throughput, low latency analytics, and accurate counting all at once.
A good interview summary is this. Ad Click Aggregator is hard because it combines write heavy ingestion, near real time aggregation, idempotency, and hot key traffic in one pipeline.
Key points
- **Lambda Architecture** — Real-time path for fast approximate aggregates. Batch path for exact billing reconciliation. Name both paths explicitly.
- **Click dedup is critical** — Each click has a UUID. Bloom filter at ingestion for fast dedup. Prevents inflated advertiser bills.
- **Flink watermarking** — Allow up to 1 minute of lateness. Bounds output latency.
- **ClickHouse for OLAP** — Columnar store optimized for aggregation queries.
- **Batch for billing** — Billing must use exact counts. Daily Spark job reconciles.
> Staff expectation: name Lambda Architecture explicitly. Real-time = fast and approximate. Batch = exact for billing.
Tradeoffs
**Stream only vs Lambda Architecture** — Stream only is approximate — not suitable for advertiser billing or revenue disputes. Lambda adds exact batch reconciliation at the cost of two pipelines. For ad revenue, Lambda is non-negotiable.
**Watermarking vs waiting indefinitely for late events** — Watermark allows a fixed lateness window (2 min). Beyond that, late events are routed to the batch path for exact reconciliation. Tradeoff: perfect accuracy vs bounded stream latency. The batch path catches everything the stream misses.
**ClickHouse vs Cassandra for aggregated storage** — ClickHouse is columnar OLAP — GROUP BY campaign_id with SUM(clicks) over billions of rows is hardware-accelerated. Cassandra is row-oriented OLTP — efficient for point lookups, slow for analytical aggregations. ClickHouse is the right choice for ad dashboards.
**Bloom filter dedup vs exact UUID check** — Bloom filter (0.01% false positive) handles 100M click UUIDs/day in 300 MB RAM — fast, cheap, good enough for real-time dashboards. For billing-critical top advertisers: exact UUID dedup via DB. Two-tier approach gives both speed and precision where it matters.
> "Lambda Architecture: real-time path for speed, batch path for accuracy. Name both paths — that's the staff-level signal."
Deep dives
#### Deep dive 1: Lambda Architecture — real-time stream + batch reconciliation
_The business requirement creates the architectural constraint: real-time approximate metrics for dashboards (advertisers want to see campaign performance now) AND exact counts for billing (advertisers dispute invoices with exact numbers). No single pipeline satisfies both_
> [!CAUTION]
> **🔴 Weak** — stream only (approximate)
>
> [!WARNING]
> **🟡 Strong** — Lambda Architecture explicitly. Speed layer (Flink): processes Kafka events in near-real-time, aggregates with 100ms batch windows, writes to ClickHouse. Results are fast but approximate (Flink checkpointing can reprocess, but windows have an allowed lateness boundary beyond which events are dropped). Batch layer (Spark): reads raw events from the event lake (S3-compatible, 90-day retention), runs daily exact aggregation. Batch results are exact, with 24-hour latency. Serving layer (ClickHouse): stores both approximate (updated every 30s by Flink) and exact (updated daily by Spark) counts. Advertisers see approximate for real-time view, exact for invoice
>
> [!TIP]
> **🟢 Staff+** — architectural point: the batch layer is not a fallback — it's a first-class part of the design that serves a different SLA requirement
#### Deep dive 2: Click deduplication — UUID + Bloom filter + exact reconciliation
_Click fraud via duplicate clicks directly translates to advertiser overbilling. Dedup must be accurate_
> [!CAUTION]
> **🔴 Weak** — check DB for duplicates
>
> [!WARNING]
> **🟡 Strong** — UUID per click event, Bloom filter at ingestion. The Bloom filter: expected 100M unique events/day, 0.01% false positive rate → 300 MB memory, acceptably small. On each click event: check Bloom filter. If not present: pass through, add to Bloom filter. If present: probable duplicate, drop the event. Bloom filter false positives (0.01%) = 10K legitimate clicks dropped per day out of 100M — acceptable for real-time dashboard. For billing-critical accuracy: the batch path uses exact dedup. The raw event log in S3 is the source of truth. Spark job: GROUP BY (click_id, UUID) and count distinct — exact dedup. Any click that appears in the raw log but not in the Bloom-filtered stream is captured in the batch reconciliation
>
> [!TIP]
> **🟢 Staff+** — the Bloom filter for a 24-hour window is reset daily. Old Bloom filters are discarded, new ones start fresh. UUID expiry aligns with the billing period
#### Deep dive 3: Hot partition handling — skewed ad traffic
> [!CAUTION]
> **🔴 Weak** — increase the total number of Kafka partitions cluster-wide
>
> [!WARNING]
> **🟡 Strong** — one viral ad campaign can generate 100× normal click volume. In a Kafka cluster partitioned by ad_id, this creates a hot partition
>
> [!TIP]
> **🟢 Staff+** — detection: monitor consumer lag per (topic, partition). When a partition exceeds a lag threshold, detect the hot ad_id causing the spike. Use a compound partition key (ad_id + random_suffix_0_to_N) to spread the load across N partitions. The Flink job handles the merge: KEY BY ad_id across multiple partitions gives correct aggregation regardless of how many partitions the ad's events are spread across. Weak answer: increase partition count. Strong answer: dynamic repartitioning. Monitor partition consumer lag per (topic, partition). When a partition exceeds a lag threshold: detect the hot ad_id causing the spike. Create additional Kafka partitions for that ad_id using a compound key (ad_id + random_suffix). The Flink job handles this: multiple partitions for the same ad_id, Flink aggregates across all partitions in a keyed stream (KEY BY ad_id → Flink handles the merge). The output is correct regardless of how many partitions the ad's events are spread across. At the ClickHouse write side: batching (Flink emits aggregated counts rather than raw events) reduces ClickHouse write amplification. For ClickHouse: ad_id is the partition key for the aggregated table — hot ad_id creates a hot ClickHouse partition. Fix: ReplicatedMergeTree with multiple replicas for hot ad_ids (ClickHouse routes reads to replicas)
_Why the deep dives connect to the scaling problem: "Write-heavy analytics pipeline with correctness requirements." Each deep dive addresses one constraint._
Interview script
1. Lambda framing.
2. "This system has a hard constraint: we need real-time approximate aggregates for dashboards AND exact counts for billing. That immediately implies Lambda Architecture."
3. "Real-time path: click events → Kafka partitioned by ad_id → Flink streaming job → tumbling windows aggregate clicks per ad per time window → write to ClickHouse."
4. "Batch path: raw click events stored in a data lake. A daily Spark job computes exact click counts and reconciles against real-time ClickHouse aggregates. Billing uses these exact counts."
5. "Click dedup: each click carries a client-generated UUID. Bloom filter check at ingestion. The Bloom filter's false positives just mean occasionally missing a dup — dramatically reduces load on the exact dedup check."
6. "I'd name this Lambda Architecture explicitly."
Whiteboard
```
+----------------------+
| Ad Placement Svc |
| returns ad, target |
| impressionId, sig |
+----------+-----------+
|
v
+---------+ click ad +--------------------+
| User | ---------------------> | Click Endpoint |
| Browser | | /click |
+----+----+ +-----+----------+---+
^ | |
| 302 redirect to advertiser | |
| | |
| | v
| | +-------------+
| | | Signature |
| | | Verification|
| | +------+------+
| | |
| | v
| | +-------------+
| | | Redis Cache |
| | | dedup by |
| | | impressionId|
| | +-------+-----+
| | |
| | duplicate?|
| | yes -> drop
| | |
| v no
| +-----------------------+
| | Kafka or Kinesis |
| | durable click stream |
| +-----+------------+----+
| | |
| | +------------------+
| | |
| v v
| +-------------------+ +-------------------+
| | Flink stream proc | | S3 data lake |
| | window by minute | | raw click archive |
| | aggregate clicks | +---------+---------+
| +---------+---------+ |
| | |
| v v
| +-------------------+ +-------------------+
| | OLAP analytics DB | <--------| Batch reconcile |
| | ClickHouse, | | Spark daily or |
| | BigQuery, etc | | hourly recompute |
| +---------+---------+ +-------------------+
| |
| v
| +-------------------+
+-------------------------| Advertiser Query |
| dashboard or API |
+-------------------+
```
The mental model is two paths. The serving path handles the user click and redirect fast, and the analytics path turns raw clicks into queryable per minute metrics.
If you were drawing this in an interview, I would keep the main story to five boxes first. User, Click Endpoint, Stream, Stream Processor, OLAP DB. Then add Redis for dedup and S3 plus batch reconcile only if the interviewer asks about idempotency or correctness.
Job scheduler (Airflow) Hard
Leader election · DAG dependencies · At-least-once
A single scheduler leader is elected via ZooKeeper/etcd. It polls PostgreSQL for due jobs and enqueues to SQS. Workers pull jobs, execute, send heartbeats. DAG dependency: a job runs only when all parent jobs have succeeded. Heartbeat timeout → requeue.
Problem
Running scheduled and dependency-based jobs reliably.
Hard parts: preventing double-scheduling, enforcing DAG dependencies, and ensuring jobs complete even when workers crash.
Failures
**Scheduler leader crashes mid-dispatch cycle**
Some jobs were enqueued but not recorded as enqueued. On leader re-election, new leader re-scans and re-enqueues them. Workers execute jobs twice.
_Fix:_ Transactional dispatch: enqueue to SQS + mark job as DISPATCHED in PG in the same transaction (using SQS FIFO with deduplication ID). Redis SETNX per job_id in workers prevents double execution even if enqueued twice.
**A long-running job (8 hours) holds a worker indefinitely, blocking queue progress**
Worker pool capacity exhausted by a few long-running jobs. Short jobs queue up and miss their schedule.
_Fix:_ Worker timeout per job type (configured separately). Separate worker pools: long-running jobs (timeout = 24h) and short-running jobs (timeout = 15min). Priority queue for time-sensitive jobs.
**DAG has a cycle (A → B → C → A)**
Scheduler loops infinitely trying to find a ready job in a circular dependency.
_Fix:_ Validate DAG topology at definition time (not at runtime). Topological sort: if a cycle is detected during sort, reject the DAG definition with a clear error. Never store invalid DAG definitions.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 100K job definitions, 10M job executions/day, avg execution time 5 min |
| Read QPS | 10M / 86400 ≈ 116 job dispatches/s — low throughput |
| Write QPS | 116 job state updates/s (PENDING → RUNNING → SUCCEEDED/FAILED) in PG — trivially low |
| Storage | Job definitions: 100K × 1KB ≈ 100 MB. Execution history: 10M/day × 500 bytes × 365 ≈ 1.8 TB/year — keep 90 days rolling. |
| Cache math | Near-term schedule: pre-compute jobs due in next 5 minutes → load into Redis sorted set (score = scheduled_time). Scheduler polls Redis instead of DB. Reduces DB query rate from 1/s to 1/5min. |
| Verdict | Throughput (116/s) is not the challenge. Correctness (no double-execution, no missed jobs, correct DAG ordering) is the hard problem. |
Design decisions
**Pull model (workers pull from queue) vs. push model (scheduler pushes to workers)**
→ Pull: workers pull from SQS
Push requires scheduler to track worker capacity and health. Pull: worker signals availability by polling. SQS handles capacity naturally — idle workers drain queue, busy workers don't pull. No scheduler-to-worker communication needed.
_Revisit when:_ Push model for tighter scheduling precision (< 1 second accuracy) where pull latency is too high.
**ZooKeeper vs. etcd vs. DB-based leader election**
→ etcd (or ZooKeeper) for leader election
DB-based election (UPDATE schedulers SET leader=1 WHERE id=? AND leader=0) works but has 1-5 second failover time and poll-based heartbeat adds DB load. etcd/ZooKeeper: dedicated coordination service, watch-based (event-driven) failover in < 1s.
_Revisit when:_ DB-based election is fine for non-critical schedulers or where adding etcd is operationally too heavy.
**Cron expression vs. interval-based scheduling**
→ Both: cron expressions for calendar-based jobs, interval for rate-based jobs
Cron expressions (0 9 * * MON = every Monday at 9 AM) cover calendar-aware scheduling. Intervals (every 15 min) are simpler for periodic jobs. Different data model: cron requires computing next_run_time from expression, interval just adds the interval.
_Revisit when:_ Unified model: always compute next_run_time and store it — abstracting over both cron and interval.
Follow-up Q&A
**How do you handle a job that consistently fails (always returns error)?**
Retry policy per job: max_retries with exponential backoff. After max_retries: mark as DEAD, move to DLQ, alert on-call. Never automatically retry indefinitely — it would starve the queue. Jobs in DEAD state are visible in UI for manual investigation and re-trigger.
**How do you implement job dependencies across different DAGs?**
Cross-DAG dependencies are dangerous — they create hidden coupling. Better pattern: use an event/message when DAG-A completes, trigger DAG-B as a separate Kafka event. Avoids circular dependency in DAG topology validation. If you must have cross-DAG deps: treat other DAG's completion as an external sensor (polling or event-based trigger).
**How do you handle jobs that need exclusive access to a resource?**
Job-level mutex: before starting execution, worker acquires a Redis lock (SETNX) on the resource_id. If lock already held: job goes back to queue with a delay. Lock TTL slightly longer than max job duration to auto-release on worker crash.
**How would you implement backfill (run past missed executions)?**
Backfill creates explicit execution instances for each missed time slot. Generates N jobs (one per missed window) and enqueues them. Worker executes with execution_date parameter = the historical date, not current time. This is how Airflow's backfill feature works.
**What's the right timeout for the heartbeat?**
Heartbeat interval = 30s. Timeout = 2× interval (60s) as minimum to avoid false positives on GC pause or slow disk. For jobs that do no I/O for long periods (pure computation): progress-based heartbeat (% complete) instead of time-based.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: end-to-end latency, consistency lag, fan-out depth. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: active-active or warm standby; conflict resolution on merge. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — Cron** — Linux cron. Simple shell scripts. No dependencies, no retry, no visibility. Works for < 100 jobs. Breaks at any scale.
**v2 — Distributed scheduler** — etcd leader election. SQS job queue. Worker pool with heartbeats. PG for job state + DAG definitions. DLQ for failed jobs. Handles 10M jobs/day.
**v3 — Enterprise** — Web UI for DAG visualization. Sensor-based external triggers. SLA monitoring and alerting. Job priority queues. Backfill support. Multi-tenant with team-level isolation.
Why it's hard to scale
The hard part in Job Scheduler is time. You are not just storing jobs. You have to find the right jobs at the right moment, execute them close to their scheduled time, and still keep the system durable when workers crash.
There are three main scaling pain points. First, time based lookup gets expensive. If you store recurring schedules like cron expressions, you cannot scan every job every second to see what is due. That is why you usually separate the job definition from execution instances and index executions by time. Second, precision and throughput fight each other. At 10k jobs per second, polling the database very frequently creates huge read load, but polling less often makes jobs late. A common fix is a two phase design where the database holds durable schedule state and a queue handles near term delivery. Third, retries and failures create duplicate work. If a worker dies mid job, the system must retry, which means you need at least once delivery and idempotent tasks so running a job twice does not break things.
A good interview summary is this. Job Scheduler is hard to scale because it combines time based querying, high throughput dispatch, and failure handling in one system. The system has to be both precise like a clock and resilient like a queue.
Key points
- **Scope it first** — Core: define jobs with schedules (cron or interval), define DAG dependencies, execute jobs reliably, retry on failure, monitor status. Out of scope unless asked: live streaming jobs, sub-second scheduling, multi-tenant isolation.
- **Single leader — no double-dispatch** — Leader election via etcd or ZooKeeper. Only the leader polls the DB for due jobs and enqueues them. Two schedulers running simultaneously = same job enqueued twice = duplicate execution. Single leader is the correct default.
- **Pull model — workers poll SQS** — Workers signal availability by pulling from SQS. SQS handles capacity naturally: idle workers drain queue, busy workers don't pull. Scheduler needs no knowledge of worker count or health. Scale workers independently.
- **Heartbeat requeue for fault tolerance** — Worker sends heartbeat every 30s. Scheduler monitors: if heartbeat stops for 60s, mark job as timed out and re-enqueue. At-least-once execution — jobs must be idempotent (check "already ran" at start).
- **DAG validation at definition time** — Topological sort on DAG definition. Cycle detected → reject with error. Never store an invalid DAG. Cycle detection at runtime is too late — jobs would loop forever without a natural termination condition.
- **Dead letter queue for permanent failures** — Max retries (e.g., 3) with exponential backoff. After max retries: move to DLQ, alert on-call. Never silently discard failed jobs. DLQ entries visible in UI for manual re-trigger after root cause fix.
- **Failure mode to name** — Leader crashes mid-dispatch: SQS deduplication ID prevents duplicate enqueue (idempotent). New leader is elected in <1s (etcd watch-based). Jobs in-flight continue executing under their existing workers — no interruption.
> Leader election + heartbeat requeue + SETNX idempotency = no double schedule, no lost jobs, no double execution.
Tradeoffs
**Single leader vs multi-scheduler** — Single leader prevents double-scheduling without coordination. Multi-scheduler requires distributed locking (etcd/ZooKeeper) for every dispatch decision. Single leader is correct until scheduling throughput itself becomes a bottleneck (rare — 116 jobs/sec is trivial).
**At-least-once vs exactly-once execution** — Exactly-once requires 2PC or distributed transactions — complex and slow. At-least-once + idempotent job logic (check "already ran" at job start) is simpler and correct. Make idempotency a platform contract, not a framework guarantee.
**Pull (workers poll SQS) vs push (scheduler dispatches)** — Pull: workers signal availability by polling — scheduler needs no knowledge of worker capacity. SQS handles capacity naturally. Push: tighter scheduling precision but scheduler must track worker health and capacity. Pull is the right default.
**DB-based leader election vs etcd/ZooKeeper** — DB election (UPDATE SET leader WHERE leader=0) works but has 30–60s failover via polling. etcd/ZooKeeper: event-driven watch, <1s failover. If etcd is already in the stack (e.g., Kubernetes cluster), use it. If not, DB election avoids adding a new dependency.
> "Leader election prevents double-scheduling. Heartbeat requeue prevents lost jobs. SETNX prevents double execution."
Deep dives
#### Deep dive 1: Leader election — preventing double-dispatch with ZooKeeper/etcd
> [!CAUTION]
> **🔴 Weak** — run multiple scheduler instances in parallel for redundancy
>
> [!WARNING]
> **🟡 Strong** — two scheduler instances running simultaneously would both scan the same job table and enqueue the same jobs twice. Workers execute jobs twice → idempotency violations, corrupted state, duplicate emails, double payments. Weak answer: use a DB lock. Strong answer: ZooKeeper or etcd leader election. etcd approach: all scheduler instances compete to create an ephemeral key /scheduler/leader with their instance ID. The TTL is 15 seconds (heartbeat interval). Only one instance can create the key — that instance is the leader. Other instances watch the key and wait. If the leader crashes: key expires in 15 seconds, election re-runs
>
> [!TIP]
> **🟢 Staff+** — DB-based election (UPDATE schedulers SET is_leader=1, heartbeat_at=now() WHERE id=? AND is_leader=0) works but has 30-60 second failover (depends on heartbeat check frequency) and adds polling load to PG. etcd/ZooKeeper: event-driven (watch-based), <1 second failover, designed for coordination. The choice is operational: if you already have etcd in your stack (Kubernetes uses it), use it. If not, DB-based is acceptable
#### Deep dive 2: DAG dependency enforcement — correct job ordering at scale
_Airflow-style DAGs: task B can only run after task A succeeds. At 10M executions/day with complex DAGs (some with 50+ tasks), the scheduler must efficiently find tasks that are ready to run_
> [!CAUTION]
> **🔴 Weak** — scan all tasks periodically
>
> [!WARNING]
> **🟡 Strong** — event-driven dependency resolution. When a task completes: publish a TASK_COMPLETED event. The scheduler consumes this event, checks if all dependencies for downstream tasks are now satisfied, and enqueues ready tasks. Dependency check: SELECT count(*) FROM task_instances WHERE dag_run_id=? AND task_id IN (upstream_tasks) AND status != 'SUCCESS'. If count = 0: all upstreams succeeded, enqueue the task
>
> [!TIP]
> **🟢 Staff+** — this check is a hotspot under high fan-out DAGs (one task → 100 downstream tasks). Batch the dependency check: on TASK_COMPLETED, add the dag_run_id to a Redis set. A low-frequency background scanner processes the set, checks all downstream tasks for that dag_run, enqueues ready ones. Reduces per-event DB queries from O(downstream_tasks) to O(1) per event. At-least-once delivery: if the dependency check fails (DB unavailable), the task remains in PENDING state. The periodic scanner catches it on the next cycle
#### Deep dive 3: Fault tolerance — heartbeat timeout, at-least-once, and idempotent workers
> [!CAUTION]
> **🔴 Weak** — mark a job as failed only when the worker explicitly reports failure
>
> [!WARNING]
> **🟡 Strong** — a worker executing a job may crash mid-execution. The job must be retried
>
> [!TIP]
> **🟢 Staff+** — at-least-once design: worker sends heartbeat every 30 seconds (UPDATE task_instances SET last_heartbeat=now() WHERE id=? AND status='RUNNING'). Scheduler scans for stale heartbeats: SELECT id FROM task_instances WHERE status='RUNNING' AND last_heartbeat < now() - INTERVAL '60s'. Re-enqueues timed-out tasks. This gives at-least-once execution — the job may run twice if the worker crashes and recovers but the heartbeat was temporarily delayed. Exactly-once requires idempotent jobs: a job that can be safely run twice must produce the same result (send-email with deduplication ID, db-insert with upsert, file-generation with atomic rename). Staff+ design principle: the scheduler guarantees at-least-once; job authors are responsible for idempotency. This is an explicit contract documented in the platform's API. For non-idempotent jobs: add an explicit "already-ran" check at job start (SELECT 1 FROM job_executions WHERE job_id=? AND execution_date=? AND status='SUCCEEDED'). Dead letter queue for jobs that fail beyond max_retries — never silently discard
_Why the deep dives connect to the scaling problem: "Time-based querying, high-throughput dispatch, failure handling." Each deep dive addresses one constraint._
Interview script
1. Leader-election script.
2. "Clarifying questions: are we building a general-purpose scheduler — cron-style plus DAG dependencies — or focused on a specific use case like ETL pipelines? And what's the job volume?"
3. "Good — general-purpose, DAG dependencies, 10M job executions/day. Core features: define jobs with schedules, define dependencies, execute reliably, retry on failure, monitor status. Out of scope: real-time streaming jobs, sub-second scheduling."
4. "The most important correctness requirement: no double-dispatch. Two schedulers running simultaneously would both enqueue the same job. Workers would execute it twice. This drives the leader election design."
5. "Leader election: etcd ephemeral key with TTL. All scheduler instances compete to create /scheduler/leader. Only one wins — that instance dispatches. Others watch the key. On leader crash: key expires in 15 seconds, re-election completes in <1 second."
6. "Dispatch: leader polls PostgreSQL for jobs due in the next 5 minutes, enqueues to SQS. Workers pull from SQS — pull model means workers signal availability naturally, no capacity tracking needed."
7. "DAG dependency: job enters the queue only when all parent jobs have status SUCCEEDED. Event-driven check: on job completion, evaluate downstream tasks. Cycle detection at DAG definition time via topological sort — reject invalid DAGs before they're stored."
8. "Heartbeat requeue: worker sends heartbeat every 30s. If heartbeat stops for 60s, scheduler re-enqueues. At-least-once execution — jobs must be idempotent. Platform contract: idempotency is the job author's responsibility, not the scheduler's."
Whiteboard
```
+-------------------+
| User |
+---------+---------+
|
POST /jobs
GET /jobs
|
v
+------------+------------+
| API Service |
+------------+------------+
|
+-----------------+-----------------+
| |
v v
+--------+--------+ +--------+---------+
| Jobs Table | | Executions Table |
| job definition | | run instances |
+--------+--------+ +--------+---------+
| |
| GSI on user_id + time
| |
| v
| +---------+---------+
| | Status Query Path |
| +-------------------+
|
| every 5 min scans next ~5 min
v
+--------+--------+
| Scheduler Cron |
| / Dispatcher |
+--------+--------+
|
| enqueue with delay
v
+--------+---------+
| Delayed Queue |
| SQS / Redis |
+--------+---------+
|
| messages become visible near run time
v
+----------+----------+------------+
| | |
v v v
+----+-----+ +----+-----+ +---+------+
| Worker A | | Worker B | | Worker N |
+----+-----+ +----+-----+ +---+------+
| | |
+----------+----------+------------+
|
| fetch job details
v
+--------+--------+
| Jobs Table |
+--------+--------+
|
| execute task
v
+--------+--------+
| Task Handler(s) |
| email, webhook, |
| cleanup, etc. |
+--------+--------+
|
+---------+----------+
| |
v v
+-------+-------+ +-------+--------+
| success | | failure |
| mark complete | | retry w backoff|
+-------+-------+ +-------+--------+
| |
+---------+----------+
|
v
+--------+---------+
| Executions Table |
| status updates |
+------------------+
```
Draw the two tables first — that is the data model. Then draw the Scheduler scanning and enqueueing. Then the worker pool. Then the success/failure split at the bottom. Save the Status Query Path and GSI for if the interviewer asks about read patterns.
Client sends a UUID idempotency key. API checks if the key exists in DB first. If yes, return cached response. If no, one ACID transaction: debit buyer, credit merchant, write webhook row to the outbox table — all in the same transaction. Async outbox worker delivers the webhook.
Problem
Processing payments reliably without double-charging.
Hard parts: idempotency (retries must not double-charge), atomic webhook delivery (outbox pattern), and double-entry accounting.
Failures
**PSP (Stripe/Adyen) times out — did the charge go through or not?**
Unknown state: Robinhood's DB shows PENDING, exchange may have charged customer.
_Fix:_ Never assume timeout = failure. Query PSP for status of the idempotency key before retrying. If PSP confirms charge: mark SUCCEEDED. If not found: retry. Idempotency key ensures PSP treats retry as same request.
**Outbox worker fails to deliver webhook after many retries**
Merchant never receives payment confirmation. Their system shows order as unpaid. Customer service nightmare.
_Fix:_ Dead letter queue for permanently failed webhooks (after N retries with exponential backoff). Alert merchant to check their endpoint. Provide webhook delivery history in merchant dashboard. Always store raw webhook payload for manual re-delivery.
**Database splits (partition) during payment processing**
Payment accepted on one DB partition, not visible on the other. Double charges on recovery.
_Fix:_ PG with synchronous replication: leader + synchronous standby. No asynchronous replica for payment writes. Synchronous replication means partition = unavailability, not inconsistency. Availability sacrifice is correct here — better to fail than to double-charge.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 1M merchants, 10K TPS peak (Black Friday), avg transaction value $50 |
| Read QPS | 10K TPS × 1 idempotency key lookup = 10K DB reads/s |
| Write QPS | 10K TPS × (1 payment + 1 outbox row + 1 event log row) = 30K DB writes/s — needs write sharding |
| Storage | Event log: 10K events/s × 300 bytes × 86400 × 365 ≈ 95 TB/year — shard by merchant_id |
| Cache math | Idempotency key cache: 10K active keys × 200 bytes = 2 MB Redis — trivial. Payment state cache: 100K active payments × 500 bytes = 50 MB — also trivial. |
| Verdict | 30K write/s hits PG's practical ceiling (~50K writes/s per node). At Black Friday peak, need write sharding by merchant_id. This is the scaling decision that matters. |
Design decisions
**Synchronous vs. async PSP call**
→ Synchronous with timeout (2s) + async reconciliation
Users expect immediate feedback ('payment accepted'). PSP async webhooks can arrive 30+ seconds later. Synchronous with a 2s timeout: if PSP responds in 2s (usually), user gets immediate confirmation. If timeout: mark PENDING, reconcile asynchronously, show user 'processing'. Best of both.
_Revisit when:_ Pure async for batch payment use cases where immediate confirmation isn't needed (B2B invoicing).
**Single currency vs. multi-currency ledger**
→ Ledger in minor units of each currency (cents, pence, etc.)
Never store monetary amounts as floating point. $1.50 in the ledger = 150 (cents). Arithmetic stays integer. Display layer divides by minor unit exponent. Currency in the event row prevents conversion errors.
_Revisit when:_ Convert to a single base currency (USD) at storage time if multi-currency reporting is required. Store both original currency and base currency amount.
**Distributed ledger vs. single PG shard**
→ Single PG shard per merchant cluster, sharded by merchant_id
Cross-shard transactions in distributed ledgers are hard (2PC). Merchant's payments are independent — no cross-merchant transactions. Shard by merchant_id: all of a merchant's payments are on one shard, enabling ACID single-shard transactions.
_Revisit when:_ Marketplace payments (split payments to multiple merchants) require cross-shard coordination — this is the hard problem in payment system design.
Follow-up Q&A
**How do you handle a refund that crosses a billing period?**
Refund is a new event in the event log: negative debit on customer account, negative credit on merchant account. It doesn't modify the original payment event (immutable). Revenue recognition in accounting handles the period crossing — that's an accounting problem, not a systems problem. Event log has all the data needed.
**How do you prevent money from being created or destroyed (double-entry integrity)?**
Invariant: sum of all debit events + sum of all credit events = 0. Run this audit query periodically (daily batch job). If non-zero: alert on-call, freeze affected accounts, investigate. This is why double-entry accounting is non-negotiable — it's self-auditing.
**How would you handle marketplace payments (buyer pays seller, platform takes a cut)?**
Three-leg transaction: debit buyer, credit seller (minus platform fee), credit platform (the fee). All three legs in one ACID transaction within one shard. If buyer and seller are on different shards: saga pattern (reserve buyer funds → credit seller → commit buyer debit). Two-phase saga with compensating transactions on failure.
**How do you prevent a merchant from issuing more refunds than they received in payments?**
Available balance check before every refund: available_balance = sum(payments) - sum(refunds) - sum(pending_refunds). If refund amount > available_balance: reject. This check is within the same ACID transaction as the refund event creation. No race condition.
**How would you implement subscription billing?**
Subscription is a scheduled payment: cron-triggered job that runs the payment pipeline on the renewal date. Idempotency key = subscription_id + billing_period. Failed charge: retry 3× over 3 days (dunning). After 3 failures: suspend subscription, notify user. Same payment infrastructure, different trigger mechanism.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: end-to-end latency, consistency lag, fan-out depth. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: active-active or warm standby; conflict resolution on merge. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — MVP** — Stripe API calls directly. No event log. Simple payment table. Works for early revenue. No audit trail, no retry safety.
**v2 — Financial correctness** — Event sourcing / double-entry ledger. Idempotency keys. Outbox pattern for webhooks. Reconciliation job. Handles SMB merchant volume.
**v3 — Enterprise scale** — Shard by merchant_id. Marketplace split payments (saga). Multi-currency. Subscription billing engine. Regulatory compliance (PCI DSS, SOC2). Fraud detection ML.
Why it's hard to scale
The hard part in a Payment System is that every request involves real money, so you need both scale and correctness at the same time. A slow feed can be annoying. A duplicated or lost payment is a business disaster.
There are three big scaling pain points. First, the write path is safety critical. At 10k plus TPS, you are creating and updating payment records fast, but you also need idempotency so retries do not double charge a customer. Second, the workflow is asynchronous because external payment networks can timeout or respond later, so your system must track uncertain states and reconcile later instead of assuming success or failure immediately. Third, durability and auditability matter much more than in a normal app. You cannot just keep the latest row state. You need a full history of what happened so you can recover, reconcile, and answer disputes.
A good interview summary is this. Payment Systems are hard to scale because they combine high write throughput, strict financial correctness, and unreliable external dependencies. You are not just processing requests quickly. You are making sure money movement is never lost, duplicated, or misreported.
Key points
- **Scope it first** — Core: accept payments, charge PSP, deliver webhooks to merchants, maintain ledger. Out of scope unless asked: subscriptions, marketplace split payments, refunds (unless asked), tax calculation.
- **Idempotency key — store BEFORE PSP call** — Client sends UUID. Server stores key as PENDING before calling PSP. On retry: find PENDING → query PSP by idempotency key → update to COMPLETED. If stored after PSP call: crash between them → retry creates duplicate charge.
- **Outbox in the same ACID transaction** — INSERT INTO payments + INSERT INTO outbox_events in one transaction. Either both commit or neither does. Async worker delivers the webhook. Eliminates lost webhooks without distributed transactions.
- **Double-entry accounting — immutable events** — Every payment = debit buyer + credit merchant in one transaction. Balance = sum(events). Never UPDATE a balance field. Invariant: sum(all debits + credits) = 0. Auditable, replayable, legally required.
- **Shard by merchant_id** — All of a merchant's payments on one shard → single-shard ACID for all their transactions, no 2PC needed. Marketplace split payments (cross-merchant) use saga pattern with compensating transactions.
- **Reconciliation job** — Nightly batch: compare internal ledger against PSP settlement report. Flag any discrepancy for manual review. Financial systems drift — reconciliation catches what the application logic missed.
- **Failure mode to name** — PSP call times out — did the charge happen? Never assume timeout = failure. Query PSP by idempotency key for status. If PSP confirms: mark COMPLETED. If not found: retry. If ambiguous: hold payment in PENDING and alert ops.
> Idempotency key = no double charges. Outbox = no lost webhooks. Double-entry = correct accounting.
Tradeoffs
**Outbox vs direct webhook call** — Direct webhook after DB commit = webhook lost if process crashes between commit and HTTP call. Outbox written in the same ACID transaction = committed atomically. Async worker delivers reliably with retries. Outbox is the only correct pattern for reliable event delivery.
**Synchronous PSP call vs async queue** — Sync PSP call is simpler and gives immediate user feedback — PSP typically responds in <500ms. Async queue adds latency for the common case. Use sync with a 2s timeout; on timeout mark PENDING and reconcile async. Best of both.
**Shard by merchant_id vs global single DB** — All of a merchant's payments on one shard enables ACID single-shard transactions — no 2PC needed. Cross-merchant transactions (marketplace split payments) are the hard case — handle with saga pattern. Never shard randomly; shard by the transaction's natural ownership boundary.
**Idempotency key stored before vs after PSP call** — Storing the key after the PSP call: if the process crashes between PSP success and DB write, retry creates a duplicate charge. Store the key (status=PENDING) BEFORE the PSP call — on retry, find PENDING key, query PSP for status, update to COMPLETED. Sequence matters.
> "Idempotency key prevents double charges. Outbox in the same transaction prevents lost webhooks. Non-negotiable in financial systems."
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
#### Deep dive 1: Idempotency — preventing duplicate charges under network failures
> [!CAUTION]
> **🔴 Weak** — check if the payment already exists before processing. At 10K TPS with network timeouts, you can't reliably distinguish "never received" from "received and failed" without an idempotency key
>
> [!WARNING]
> **🟡 Strong** — client generates UUID before sending any payment request. Server stores the key with status=PENDING before calling the PSP. On retry: find PENDING key → query PSP by the same idempotency key → update to COMPLETED or FAILED
>
> [!TIP]
> **🟢 Staff+** — sequence detail: storing the key AFTER the PSP call is a critical mistake. If the process crashes between PSP success and DB write, the retry has no key to find — it creates a new charge. Store the key BEFORE the PSP call unconditionally. On retry: find PENDING key, call PSP with same idempotency key (PSP deduplicates on its end too), update status. The entire chain is idempotent end-to-end. This sequence is the single most important correctness detail in payment system design
#### Deep dive 2: Outbox pattern — guaranteed webhook delivery without distributed transactions
> [!CAUTION]
> **🔴 Weak** — after recording the payment in the DB, make an HTTP call to deliver the webhook. If the process crashes between DB commit and HTTP call, the webhook is lost. The merchant never learns about the payment
>
> [!WARNING]
> **🟡 Strong** — outbox pattern. In the same ACID transaction that records the payment: INSERT INTO outbox (event_type, payload, status). Either both the payment and the outbox entry commit, or neither does. An async worker polls the outbox for PENDING entries and delivers them with retries and exponential backoff
>
> [!TIP]
> **🟢 Staff+** — idempotency for webhooks: the webhook payload must include an idempotency key. The merchant's endpoint may receive the same webhook multiple times (delivery retry after a timeout). If the merchant's system isn't idempotent, a payment can be processed twice on their end. Document this contract explicitly: webhook delivery is at-least-once, merchant endpoints must be idempotent
#### Deep dive 3: Multi-step payment flow — saga pattern for distributed transactions
> [!CAUTION]
> **🔴 Weak** — use a distributed transaction (2PC) across shards to atomically debit the buyer and credit the seller. 2PC is slow, blocks resources during the coordinator phase, and is prone to blocking on coordinator failure
>
> [!WARNING]
> **🟡 Strong** — saga pattern. A saga is a sequence of local transactions, each with a compensating transaction on failure. Payment saga: (1) RESERVE buyer funds (local ACID transaction), (2) CREDIT seller, (3) CAPTURE buyer reservation. On failure at step 2: run compensating transaction for step 1 (release reservation)
>
> [!TIP]
> **🟢 Staff+** — durability: the saga state machine must be stored durably. If the saga orchestrator crashes mid-saga, it must resume from the last committed step on restart — not restart from the beginning (which would double-charge). Store saga state in PostgreSQL with the current step and status. On restart: read uncommitted sagas, resume from last committed step. Sagas are eventually consistent — the DB may briefly be in an intermediate state, but all failures are handled gracefully by the compensating transactions
_Why the deep dives connect to the scaling problem: "High write throughput, strict financial correctness, unreliable external dependencies." Each deep dive addresses one constraint._
Interview script
1. Financial-integrity-first script.
2. "Before I start: are we designing a payment processor like Stripe — where merchants integrate via API — or an end-user checkout flow? And do we need to handle marketplace split payments?"
3. "Good — API-first like Stripe, single merchant for now. Core features: accept payment, call PSP, deliver webhook to merchant, maintain ledger. Out of scope: subscriptions, refunds, multi-currency (unless asked)."
4. "Scale: 10K TPS peak (Black Friday). The hard constraint isn't throughput — it's correctness. A slow feed is annoying. A duplicate charge is a legal problem."
5. "Two non-negotiables I'd state upfront: idempotency and reliable webhook delivery. These are the failure modes that matter in production."
6. "Idempotency: client generates a UUID before sending any payment request. Server checks if this key exists before processing. If yes, return the cached response — no new charge. Key detail: store the idempotency record BEFORE calling the PSP. If stored after and the process crashes in between, retry creates a duplicate charge."
7. "Outbox pattern: in the same ACID transaction that records the payment, write a row to an outbox table. An async worker reads the outbox and delivers the webhook with retries and exponential backoff. Either both the payment and the outbox entry commit, or neither does — no lost webhooks."
8. "Ledger design: double-entry event sourcing. Every payment = debit buyer + credit merchant, both in one transaction. Balance is never a mutable field — it's always derived from the sum of events. Required for regulatory compliance."
9. "Sharding: by merchant_id. All of one merchant's payments on one shard — enables single-shard ACID, no distributed transactions needed."
Whiteboard
```
+--------------------+
| Merchant Backend |
| uses API keys |
+---------+----------+
|
| HTTPS
v
+-------------+ +-------+--------+
| Customer | | API Gateway |
| Browser | | auth, routing, |
| checkout UI | | rate limiting |
+------+------+ +---+---------+---+
| | |
| card entry via | |
| hosted iframe / SDK | |
v | |
+------+-----------------------+ |
| Secure Payment SDK / iFrame | |
| card data goes to processor | |
+--------------+---------------+ |
| |
| encrypted card data |
| |
v v
+--------+---------+ +--------+---------+
| Transaction | | PaymentIntent |
| Service | | Service |
| creates charge | | create/read |
| records | | payment intent |
+----+--------+----+ +----+--------+----+
| | | |
| +----------------+ |
| read/write |
v v
+---------------------------------------------+
| Operational Database |
| merchants, payment_intents, transactions, |
| attempts, statuses |
+-------------------+-------------------------+
|
| CDC from DB log
v
+--------+---------+
| Kafka / Event |
| Stream |
| immutable events |
+---+----+----+----+
| | |
| | |
| | +------------------+
| | |
| v v
| +---------+ +-------------+
| | Audit | | Webhook |
| | Service | | Service |
| | history | | notify |
| +----+----+ | merchants |
| | +------+------+
| | |
| v | HTTPS POST
| +---------+ v
| | Cold | +-------------+
| | Storage | | Merchant |
| | S3 etc | | Webhook URL |
| +---------+ +-------------+
|
v
+-------+--------+
| Reconciliation |
| Service |
| resolves |
| timeouts |
+-------+--------+
|
| query status / batch files
v
.---------------------------------------.
| External Payment Networks and Banks |
| Visa, Mastercard, issuing banks |
'---------------------------------------'
```
The mental model is simple. PaymentIntent Service manages the customer payment lifecycle, Transaction Service talks to the outside payment world, and the database plus CDC plus Kafka gives you a durable history so you do not lose money movement events.
If you are drawing this in an interview, you can start with just five boxes. Merchant, API Gateway, PaymentIntent Service, Transaction Service, Database. Then add CDC, Kafka, Reconciliation, and Webhooks only if the interviewer pushes on durability or async safety.
Host agents batch metrics and push every 10s to Kafka. Three consumers: TSDB write path, Flink alert evaluator, and Rollup Worker. The TSDB is columnar and time-partitioned. Rollup: 1s samples → 1min averages (30 days) → 1hr averages (forever). Cardinality explosion: each unique metric + label combination creates a new series.
Problem
Collect and use system health data at very large scale. Ingest measurements, store as time series, query in dashboards, and trigger alerts.
The hard part: huge write volume, fast queries across long time ranges, reliable alerting, and cardinality explosion.
Failures
**Cardinality explosion: a developer adds user_id as a metric tag**
1M active users × 100 metrics each = 100M new series created in hours. TSDB memory exhausted. Query performance degrades catastrophically.
_Fix:_ Cardinality enforcement at ingestion: count distinct tag combinations per metric. Alert when a metric's cardinality exceeds 10K series. Reject (or quarantine) metrics that exceed the cardinality cap with an actionable error message to the developer.
**Alert evaluator falls behind during an incident**
Alerts that should fire during the incident are delayed. On-call engineer doesn't get paged. Incident duration extends.
_Fix:_ Alert evaluation is the highest-priority consumer. Dedicated Kafka consumer group for alerts with separate scaling from dashboard consumers. Alert evaluation SLA: P99 < 30s. Monitor alert lag as a first-class metric.
**TSDB write node goes down during peak ingestion**
5M points/s cannot be written. If Kafka buffer is only 1 hour, unprocessed metrics are lost.
_Fix:_ Kafka retention: 24 hours. This gives 24h to recover TSDB before data loss. TSDB primary + synchronous replica. On primary failure: promote replica (< 30s). Replay from Kafka after recovery. Dashboards may be briefly stale — acceptable.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 500K hosts, 200 metrics/host, push every 10s, 1M alert rules evaluated every 60s |
| Read QPS | 500K × 200 / 10 = 10M data points/s ingested → Kafka → TSDB |
| Write QPS | Dashboard queries: 100K users × 5 queries/min / 60 = 8,333 dashboard QPS — served from TSDB + cache |
| Storage | 10M points/s × 8 bytes × 86400 = 6.9 TB/day raw. With rollup: 90% reduction after 24h → 690 GB/day avg. Over a year: ~250 TB. |
| Cache math | 10M series × avg active series metadata 100 bytes = 1 GB series index in memory per TSDB node (10 nodes = 10 GB total for the index). Feasible. |
| Verdict | 10M writes/s is the real scale challenge. Single TSDB node handles ~500K writes/s. Need 20 TSDB nodes minimum. Kafka's role as the durable buffer (24h retention) is critical. |
Design decisions
**Push vs. pull for metric collection**
→ Push (agents push to ingestion service) as default, pull (Prometheus-style) for some use cases
Push: simpler agent, no firewall rules needed from collector to target, better for ephemeral containers. Pull: collector controls sampling rate, easier to detect down targets. Datadog uses push. Prometheus uses pull. For interview: state both exist, default to push for cloud-native environments.
_Revisit when:_ Pull for service health checks (if target doesn't send a metric, collector can detect it's down). Push for all other metrics.
**TSDB choice: InfluxDB vs. TimescaleDB vs. Prometheus vs. M3DB**
→ InfluxDB or M3DB (purpose-built for high-cardinality time-series at Datadog scale)
Prometheus: excellent for Kubernetes monitoring but limited long-term storage and single-node. TimescaleDB: excellent SQL support but lower write throughput. InfluxDB/M3DB: designed for 10M+ writes/s with high cardinality.
_Revisit when:_ TimescaleDB if SQL query flexibility is needed for complex analytics on top of metrics.
**Alert evaluation: polling vs. streaming**
→ Polling (scheduled query every 30-60s) as default
Streaming (Flink/Kafka Streams): lower latency (<5s) but much higher operational complexity. Most monitoring alerts don't require sub-minute latency. Polling is simpler, testable, and predictable.
_Revisit when:_ Streaming for anomaly detection use cases where <60s detection time matters (e.g., real-time fraud, SLO burn rate alerting).
Follow-up Q&A
**How do you handle a host that stops sending metrics?**
Absence detection: alert on 'metric not received in last N intervals'. Implementation: each metric has a last_received timestamp. Background job checks all active series every 30s, fires 'host down' alert if any series hasn't updated. This is a separate alert type from threshold-based alerts.
**How do you make dashboards feel fast even for 30-day time ranges?**
Rollup architecture: raw 1s data retained for 24h. 1min aggregates for 30 days. 1hr aggregates forever. Dashboard query selects the appropriate rollup based on time range. 30-day dashboard: uses 1min rollups (43,200 points) vs. raw (2.6M points). 10× faster query.
**How do you handle high cardinality without completely blocking users?**
Graduated enforcement: warn at 1K series, soft cap at 10K (logs + alert), hard cap at 100K (reject with actionable error). Allow overrides for trusted teams with explicit justification. Provide tooling to identify which tags are causing the explosion. Don't just block — help users fix it.
**How do you ensure alert reliability during an incident?**
Separate alert evaluation pipeline from dashboard query pipeline. During an incident, users flood dashboards — this must not compete with alert evaluation. Separate Kafka consumer groups, separate compute. Alert evaluation gets reserved CPU quota. Circuit breaker: if TSDB is overloaded, serve alerts from a pre-computed alert state cache.
**How would you add distributed tracing to this system?**
Tracing is a different data model: tree of spans with parent-child relationships. Store traces in a columnar store (Cassandra or ClickHouse) partitioned by trace_id. Sampling is critical — storing every trace at 100K requests/s is expensive. Head-based or tail-based sampling at 0.1-1%. Separate from metrics pipeline but can share Kafka infrastructure.
**What metrics and alerts would you put on this system?**
Track golden signals: latency p50/p99 per API, error rate, saturation (CPU, queue depth, cache hit ratio). Business metrics: end-to-end latency, consistency lag, fan-out depth. Alert on SLO burn — e.g. p99 redirect latency >200ms for 5min, cache hit ratio drop below 90%, or write failure rate spike. Dashboard per service with dependency health.
**How would you test and roll out changes safely?**
Contract tests on APIs, load tests on read/write hot paths, chaos tests on Redis/DB failures. Shadow traffic for risky changes (new ranking, new ID scheme). Feature flags for incremental rollout. Canary 1% → 10% → 100% with automatic rollback on error-rate regression.
**How do you handle a regional outage or disaster recovery?**
Multi-AZ by default; multi-region for critical paths. Define RPO/RTO: active-active or warm standby; conflict resolution on merge. Async replication to secondary region; DNS/geo routing failover. Run game days. Document degraded mode — what features drop vs what must stay up.
Evolution
**v1 — Single host** — StatsD + Graphite. Single TSDB node. Dashboard is a couple of time-series charts. Works for one team monitoring one service.
**v2 — Scale ingestion** — Kafka for ingestion buffering. 10-node TSDB cluster. Rollup pipeline. Alert evaluation service. Cardinality caps. Handles 500K hosts.
**v3 — Enterprise** — M3DB or purpose-built TSDB for 10M writes/s. Distributed tracing integration. ML anomaly detection. Cross-service SLO tracking. Custom metrics plugins. Multi-tenant with per-team quotas.
Why it's hard to scale
The hardest part is cardinality explosion. In a metrics system, every unique combination of metric name and labels creates a new time series, so a few extra labels can turn one metric into millions of series very quickly.
That causes three scaling problems. First, ingestion gets expensive because the system is not just appending values. It also has to track metadata and indexes for huge numbers of series. Second, queries get slower because dashboards often need to scan and aggregate across many series over long time ranges. Third, alerts add pressure because they need fresh enough data and reliable evaluation even while the write path is constantly busy.
A good short interview answer is this. Metrics Monitoring is hard to scale because it combines a massive continuous write stream, expensive time range queries, and exploding series count from labels, all while the system itself needs to stay available during incidents.
Key points
- **Ingest path is write-heavy** — Agents batch metrics. Kafka buffers spikes and decouples ingestion from storage.
- **TSDB for storage** — Metrics are append-only, queried by time range. TSDB is the right default.
- **Query path is read-heavy** — Rollups + caching make dashboards fast. Rollups: raw 24h → 1min 30d → 1hr forever.
- **Alert path must be reliable** — Polling every 30–60s is the simple default. Dashboards can be stale. Alerts must not be lost.
- **Cardinality explosion** — Each unique metric + label combination = a new time series. A label like user_id creates billions of series. Enforce label allowlists and caps.
> "I'd design it as a pipeline from agents → Kafka → TSDB, then split into a query path for dashboards and an alert path for scheduled checks. The key scaling risk is cardinality explosion — control labels and use rollups for efficient reads."
Tradeoffs
**Polling alerts vs stream processing** — Polling every 30–60s is simpler and good enough for most monitoring. Stream processing gives lower latency but adds operational complexity.
**Raw data vs rollups** — Raw gives accuracy but makes long time-range queries too slow. Rollups make dashboards fast but lose detail. Both are needed.
**Accept all labels vs cardinality controls** — Flexible labels give visibility but create too many unique series. Allowlists or caps protect the system but limit visibility.
> "Start with agents, Kafka, a time-series database, rollups, and polling-based alerts. Main tradeoffs: freshness vs complexity, query flexibility vs cost, label flexibility vs cardinality explosion."
Deep dives
The three deep dives that matter most for this system, ordered by what interviewers probe hardest.
#### Deep dive 1: Cardinality explosion — the hidden scaling killer
> [!CAUTION]
> **🔴 Weak** — accept all metrics with any label combinations — flexibility is good, users know what they need
>
> [!WARNING]
> **🟡 Strong** — cardinality enforcement at ingestion. Each unique combination of metric name + label values creates a new time series. One developer adds user_id to a request_latency metric: 10M users × 1 metric = 10M new series overnight. TSDB memory exhausted, query performance collapses. Hard cap per metric (10K series), alert at 1K series, reject above cap with an actionable error
>
> [!TIP]
> **🟢 Staff+** — label allowlist: developers define allowed labels per metric at registration time. user_id is not in the allowlist by default — it requires explicit review. Provide a cardinality dashboard so developers can see the impact of their instrumentation choices before hitting the cap. The allowlist is the proactive control; the hard cap is the safety net
#### Deep dive 2: Query performance — rollups and columnar storage for time-range queries
> [!CAUTION]
> **🔴 Weak** — store raw metrics at 1-second resolution forever, query from raw data for all dashboard requests. A 30-day dashboard chart at 1-second resolution = 2.6M data points per metric per host. At 500K hosts, this is 1.3 trillion data points per query — impossible
>
> [!WARNING]
> **🟡 Strong** — multi-resolution rollup architecture. Raw (1s): retain 24 hours. 1-minute rollup: retain 30 days. 1-hour rollup: retain forever. Dashboard query routing: time range > 24h → 1-minute rollups; > 30 days → 1-hour rollups. A 30-day chart at 1-min resolution = 1,440 points vs 2.6M raw — 1800× fewer data points, no visible chart difference at typical widths
>
> [!TIP]
> **🟢 Staff+** — implementation: TimescaleDB continuous aggregates or InfluxDB tasks compute rollups automatically on insert. The rollup is always up-to-date without a separate batch job. ClickHouse for the rollup serving layer: columnar compression and SIMD-accelerated GROUP BY queries make analytical aggregations over millions of series fast
#### Deep dive 3: Alert evaluation — reliability and isolation from dashboard traffic
> [!CAUTION]
> **🔴 Weak** — run alert evaluation as a query against the same TSDB that serves dashboards. During an incident, dashboard traffic spikes 3×. Alert evaluation competes for TSDB resources and gets delayed precisely when alerts are most critical
>
> [!WARNING]
> **🟡 Strong** — dedicate separate compute for alert evaluation — separate Kafka consumer group, separate TSDB read replicas, reserved CPU quota. Alert evaluation must never compete with dashboard queries
>
> [!TIP]
> **🟢 Staff+** — s: (1) alert evaluation falls behind (Kafka consumer lag) → autoscale alert evaluation workers, alert at >30s lag; (2) TSDB replica goes down → circuit breaker switches to alternate replica; (3) flapping alerts — cooldown periods (alert must be in firing state for N consecutive evaluations before paging) prevent noise during unstable incidents. Monitor the monitoring system: alert evaluation latency is itself a first-class SLA metric
_Why the deep dives connect to the scaling problem: "Massive write stream, expensive time-range queries, exploding series count." Each deep dive addresses one dimension._
Interview script
1. Ingest-first, cardinality-aware script.
2. "Clarifying questions: are we monitoring infrastructure metrics like CPU and memory, or also custom application metrics and distributed tracing? And what are the target SLAs — how fresh do dashboards need to be, how fast do alerts need to fire?"
3. "Good — infrastructure plus custom app metrics. Dashboard freshness: 30 seconds is fine. Alert latency: under 1 minute. No tracing in scope for now."
4. "Scale: 500K hosts, 200 metrics each, push every 10 seconds = 10M data points per second. That's the number that drives everything."
5. "I'd lead with the most dangerous anti-pattern: cardinality explosion. Every unique combination of metric name and label values creates a new time series. One developer adds user_id as a label on a request latency metric: 10M users × 1 metric = 10M new series overnight. TSDB memory exhausted, queries collapse. I'd design cardinality enforcement in at ingestion — hard cap per metric, alert before the cap."
6. "Ingest path: agents batch metrics locally and push every 10s. Kafka buffers the spike and decouples ingestion from storage. 10M points/sec is within Kafka's range with appropriate partitioning."
7. "Three consumers from Kafka: (1) TSDB writer — InfluxDB or TimescaleDB, append-only, time-partitioned; (2) Alert evaluator — polls TSDB every 30-60s, isolated from dashboard traffic; (3) Rollup worker — 1s data → 1min aggregates → 1hr aggregates."
8. "Query path: rollups make dashboard queries fast. A 30-day chart at 1-min resolution = 1,440 points vs 2.6M raw points. The rollup is pre-computed — query hits the rollup table directly."
9. "Key isolation point: alert evaluation must be on dedicated compute, isolated from dashboard queries. During an incident, dashboard traffic spikes 3×. If alerts compete with dashboards for TSDB resources, alerts get delayed exactly when they're most critical."
Whiteboard
```
+----------------------+
| Users / Engineers |
+----------+-----------+
|
v
+----------------------+
| Dashboard / Query UI |
+----------+-----------+
|
v
+----------------------+
| Query Service |
| parse DSL, auth, |
| cache, query split |
+----+------------+----+
| |
cache hit | | query raw or rollups
v v
+---------+ +-------------------+
| Redis | | Time Series DB |
| Cache | | raw + rollups |
+---------+ | sharded + replica |
+---------+---------+
^
|
+---------+---------+
| Storage Consumers |
| batch writes |
+---------+---------+
^
|
+-------------+ +-------------------+ |
| Servers and | ---> | Local Agent / | ---> |
| Services | | Collector | |
| emit metrics| | buffer + batch | |
+-------------+ +---------+---------+ |
| |
v |
+-------------------------+
| Ingestion Service |
| validate, normalize, |
| auth, rate limit |
+-----+-------------+-----+
| |
| v
| +----------------------+
| | Cardinality Guard |
| | policy check |
| | label allowlist |
| +----+------------+----+
| | |
| | v
| | +-----------+
| | | Postgres |
| | | Policies |
| | | Alert cfg |
| | +-----------+
| v
| +-----------+
| | Redis |
| | series set|
| | counters |
| +-----------+
|
v
+----------------------+
| Kafka |
| durable buffer |
| partitioned stream |
+----+-------------+---+
| |
| |
| +----------------------+
| |
v v
+----------------------+ +----------------------+
| Storage Consumers | | Alert Evaluator |
| write to TSDB | | poll rules, query |
+----------------------+ | TSDB every 30 to 60s |
+----------+-----------+
|
v
+----------------------+
| Alert Events |
| firing or resolved |
+----------+-----------+
|
v
+----------------------+
| Notification Service |
| dedupe, grouping, |
| silence, escalation |
+----+----------+------+
| |
| |
v v
+---------+ +----------+
| Slack | | PagerDuty|
+---------+ +----------+
|
v
+------+
|Email |
+------+
```
The main story is ingest, buffer, store, query, then alert. If you are drawing this in an interview, I would keep the first pass even simpler with agents, ingestion, Kafka, time-series DB, query service, alert evaluator, and notification service. Then add cardinality control, cache, and rollups only if the interviewer pushes on scale or latency.
Notification system (APNs/FCM) Medium
Multi-channel · Fan-out · Dedup · Rate limits
KafkaAPNsFCMTwilioSendGridDedupDLQ
Triggers hit the Notification API, which checks user preferences, applies per-user rate limits, and deduplicates via Redis SETNX on event_id. Valid events publish to per-channel Kafka topics. Channel workers call APNs/FCM/SMS/email providers with exponential backoff and DLQ for permanent failures.
Problem
Send millions of notifications daily across push, SMS, and email with delivery guarantees, user preference respect, and survival of viral fan-out spikes.
Hard parts: multi-channel routing, at-least-once deduplication, and third-party provider rate limits during events.
Failures
**APNs returns BadDeviceToken**
Worker retries dead token forever, wasting capacity.
_Fix:_ Delete token on first BadDeviceToken. Subscribe to APNs feedback service for batch cleanup.
**Duplicate send after worker crash**
User receives two identical push notifications.
_Fix:_ Redis SETNX on event_id before send. Provider-level collapse_key as second layer.
**10M notifications in 60 seconds**
Provider rate limits hit. Queue depth grows. Delivery delayed hours.
_Fix:_ Stagger enqueue. Scale workers on consumer lag. Group non-critical notifications.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 100M users, 5 notifications/user/day, peak viral event 10M in 60s |
| Read QPS | Steady: 100M×5/86400 ≈ 5,800 deliveries/s |
| Write QPS | Peak: 10M/60 ≈ 167K enqueue/s to Kafka |
| Storage | 500M notifications/day × 500B ≈ 250 GB/day Kafka retention (7d ≈ 1.75 TB) |
| Cache math | Dedup set: 500M event_ids × 20B × 2d TTL ≈ 20 GB Redis |
| Verdict | Steady state is easy. Peak needs autoscaling workers and staggered fan-out. |
Design decisions
**Check preferences before or after queue?**
→ Before enqueue
Opted-out users never enter the queue. Preferences cached in Redis for sub-ms checks.
_Revisit when:_ After queue only if preference service is too slow — rare.
**One Kafka topic or per-channel?**
→ Per-channel topics
Independent scaling and retry policies per channel type.
_Revisit when:_ Single topic with channel header if ops simplicity matters more.
**Transactional vs marketing priority**
→ Separate queues / topics
Password resets must not wait behind marketing campaigns.
_Revisit when:_ Single queue with priority field if volume is low.
Follow-up Q&A
**How do you deliver to an offline mobile user?**
Retry until APNs/FCM acknowledges receipt (not device delivery). Providers queue latest notification for offline devices. Our job ends at provider ACK.
**How do you implement notification grouping ('5 new likes')?**
Buffer social notifications 15 min. Redis sorted set pending:{user}:{type}. Scheduled job flushes grouped message. Security alerts: delay=0, no grouping.
**How do you handle GDPR deletion?**
Delete tokens. Redis blocklist for deleted user_ids checked at every worker. Provider unsubscribe APIs for email. Document in-flight message SLA.
**How do you prevent notification storms to one user?**
Per-user hourly cap in Redis INCR with TTL. Critical channel bypasses cap. Excess marketing notifications dropped or deferred.
**How do you support scheduled notifications?**
Delayed Kafka message or Redis sorted set scored by send_time. Scheduler publishes when due. Idempotency key includes scheduled slot.
**How do you test without spamming real users?**
Sandbox provider credentials. Feature flag per user for test mode. Shadow queue that logs but does not send.
**How do you measure delivery success?**
Track enqueued → sent → provider_ack → opened. Consumer lag on Kafka as health metric. DLQ depth alerts.
**How would you add in-app notification inbox?**
Persist notification row in Cassandra/Postgres on enqueue. Push is best-effort delivery; inbox is source of truth for history.
Evolution
**v1 — Sync HTTP** — Call APNs inline. Works to ~100/s. Blocks request thread.
**v2 — Async queue** — Kafka per channel. Workers with retry + DLQ. Handles 10K/s.
**v3 — Enterprise** — Dedup, preferences, priority queues, grouping, analytics, GDPR blocklist. Handles viral spikes.
Why it's hard to scale
The hard part is fan-out: one viral event creates millions of deliveries, but third-party providers cap throughput. Kafka handles ingestion; workers and staggered delivery handle the bottleneck.
Key points
- **Scope** — Core: send push/SMS/email, respect opt-outs, retry failures, track delivery status. Out of scope unless asked: in-app inbox, rich media templates, A/B testing.
- **Async by default** — API enqueues and returns immediately. Never block the caller on APNs/FCM latency.
- **Per-channel Kafka topics** — ios-push, android-push, sms, email — independent worker pools and retry policies.
- **Dedup on event_id** — SETNX event_id with 24h TTL before send. Prevents duplicate notifications on Kafka replay.
- **Preferences before queue** — Cache opt-outs in Redis. Filter before publishing — never waste queue capacity on opted-out users.
- **Two-tier priority** — Transactional (password reset, purchase) bypasses marketing rate limits. Marketing respects max N/user/hour.
- **Invalid token cleanup** — On APNs BadDeviceToken: delete token immediately. Never retry dead endpoints.
> Prefs first, dedup second, per-channel workers third. Say those three and you cover reliability.
Tradeoffs
**At-least-once vs exactly-once** — At-least-once + Redis dedup is the production default. Exactly-once needs Kafka transactions — slower and rarely worth it for notifications.
**Single topic vs per-channel topics** — Per-channel topics let you scale iOS workers independently from email workers and apply different retry policies.
**Push vs pull delivery** — Mobile notifications must be push (APNs/FCM). Pull polling drains battery and adds latency.
**Sync send vs async queue** — Sync couples user request latency to Twilio/APNs. Async queue is always correct for notifications.
> "At-least-once delivery with dedup, preferences checked before enqueue, and staggered fan-out for viral events."
Deep dives
#### Deep dive 1: Multi-channel routing and independent scaling
> [!CAUTION]
> **🔴 Weak** — one worker pool handles all channels
>
> [!WARNING]
> **🟡 Strong** — channel-specific workers because rate limits, payload formats, and failure modes differ. APNs uses device tokens and certificate auth; email uses SMTP/API with bounce handling; SMS has per-country regulations
>
> [!TIP]
> **🟢 Staff+** — partition Kafka by user_id hash for parallelism while keeping per-user ordering within a channel. Critical notifications use a dedicated high-priority topic with reserved worker capacity
#### Deep dive 2: Deduplication under at-least-once delivery
_Worker crashes after sending but before committing offset → message redelivered → duplicate notification. Redis SETNX on event_id before send. If key exists, skip. TTL = 24h covers replay window_
> [!CAUTION]
> **🔴 Weak** — Retry until delivery succeeds — duplicates are rare.
>
> [!WARNING]
> **🟡 Strong** — Worker crashes after sending but before committing offset → message redelivered → duplicate notification. Redis SETNX on event_id before send. If key exists, skip. TTL = 24h covers replay window
>
> [!TIP]
> **🟢 Staff+** — include idempotency key in provider request (FCM collapse_key) so the provider also deduplicates. Document contract: delivery is at-least-once; consumers must be idempotent
#### Deep dive 3: Viral fan-out and provider rate limits
_One event → 10M notifications in 60s. Kafka absorbs the write spike, but APNs/FCM rate-limit per certificate. Stagger enqueue over 60–120s. Monitor provider 429 responses and backoff globally. Batch similar notifications (5 new likes → one grouped push)_
> [!CAUTION]
> **🔴 Weak** — Push to every device synchronously from the API handler.
>
> [!WARNING]
> **🟡 Strong** — One event → 10M notifications in 60s. Kafka absorbs the write spike, but APNs/FCM rate-limit per certificate. Stagger enqueue over 60–120s. Monitor provider 429 responses and backoff globally. Batch similar notifications (5 new likes → one grouped push)
>
> [!TIP]
> **🟢 Staff+** — APNs coalesces offline notifications — only latest is delivered. For badge counts, send silent data push that triggers app to fetch true count from API
#### Deep dive 4: GDPR and user deletion
_Delete device tokens immediately. Publish user_deleted event. All workers check Redis blocklist before send. Kafka messages for deleted users cannot be erased — skip at dispatch. Document 72h purge SLA for compliance_
> [!CAUTION]
> **🔴 Weak** — Delete the user row — async workers will stop eventually.
>
> [!WARNING]
> **🟡 Strong** — Delete device tokens immediately. Publish user_deleted event. All workers check Redis blocklist before send. Kafka messages for deleted users cannot be erased — skip at dispatch. Document 72h purge SLA for compliance
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Three-problem script.
2. "Core features: deliver push, SMS, and email; respect opt-outs; retry transient failures; surface delivery status."
3. "Architecture: API → preference check → dedup → Kafka per channel → channel workers → APNs/FCM/Twilio/SendGrid."
4. "Dedup: SETNX event_id in Redis before every send. At-least-once from Kafka is fine if workers are idempotent."
5. "Preferences: check Redis opt-out cache before enqueue — never queue notifications users opted out of."
6. "Viral events: stagger fan-out, monitor provider rate limits, group low-priority notifications."
7. "Invalid tokens: remove immediately on provider error. Run daily feedback sweep for batch cleanup."
Whiteboard
```
+------------------+
| Trigger sources |
| txn · marketing |
+--------+---------+
|
v
+------------------+
| Notification API |
| prefs · dedup |
+--------+---------+
|
+--------------+--------------+
| | |
v v v
+-----------+ +-----------+ +-----------+
| Kafka iOS | |Kafka SMS | |Kafka Email|
+-----+-----+ +-----+-----+ +-----+-----+
| | |
v v v
+-----------+ +-----------+ +-----------+
| iOS worker| | SMS worker| |Email worker|
+-----+-----+ +-----+-----+ +-----+-----+
| | |
v v v
APNs/FCM Twilio SendGrid
```
Interview version: API checks prefs and dedup, publishes to Kafka, workers call providers. Add DLQ and token cleanup if pushed on reliability.
Search autocomplete (Google) Medium
Trie · Top-K cache · Batch rebuild · Debounce
TrieRedisMapReduceCDNTop-KDebounce
Query logs aggregate nightly via MapReduce into frequency counts. A trie builder stores top-10 completions per prefix in Redis (prefix → JSON array). Keystrokes hit the API after 100ms debounce; CDN caches the hottest prefixes.
Problem
Return top-10 search completions within 100ms as the user types, at billions of queries per day.
Hard parts: sub-100ms latency, fresh trending terms, and trie size small enough to serve from cache.
Failures
**Trie rebuild fails at 70%**
Live trie is weeks old. Missing trending terms.
_Fix:_ Shadow build, validate, atomic flip. Rollback = flip back.
**Viral term not in trie**
Breaking news queries return empty suggestions.
_Fix:_ Hot-term detector injects temporary entries.
**'th' shard hotspot**
English prefixes skew traffic to one shard.
_Fix:_ Sub-shard by second/third character.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 10B queries/day, 5 keystrokes/query, 100ms SLA |
| Read QPS | 10B×5/86400 ≈ 578K keystroke QPS; ~58K after debounce |
| Write QPS | Nightly log ingest 1 TB/day; weekly trie rebuild batch |
| Storage | Trie ~500 MB Redis for 1M prefixes |
| Cache math | CDN top 10K prefixes → ~90% hit rate |
| Verdict | Backend QPS modest after debounce + CDN. |
Design decisions
**Rebuild frequency**
→ Weekly full + 5min hot-term injection
Captures stable trends weekly; injection handles breaking news.
_Revisit when:_ Daily rebuild for news-heavy products.
**K suggestions per node**
→ K=10
UI shows at most 10; storing more wastes space.
_Revisit when:_ K=20 if post-rank filtering removes many.
**Personalization approach**
→ Global trie + query-time re-rank
Per-user trie at 1B users is impossible.
_Revisit when:_ Per-user trie only for small enterprise search.
Follow-up Q&A
**How does Google differ from this design?**
Same trie + top-K retrieval; Google adds ML re-ranking, location, freshness, and A/B testing between trie and ranker.
**How do you handle multiple languages?**
Separate trie per language in Redis namespace en:prefix, ja:prefix. Detect from Accept-Language and script.
**How do you add spell correction?**
Symspell parallel branch: if exact trie returns <5 results, fuzzy lookup within edit distance 2 within remaining latency budget.
**How do you filter inappropriate suggestions?**
Blocklist applied at trie build time. Human review queue for flagged terms.
**How do you A/B test ranking changes?**
Cohort flag routes to different re-ranker version. Trie unchanged; only ranking layer varies.
**How do you handle CJK input?**
Character n-gram trie instead of word-based. Segment Japanese input before lookup.
**How do you update without downtime?**
Build new Redis key namespace v2:, flip routing config, delete v1 after TTL.
**How do you measure quality?**
Track suggestion CTR, zero-result rate, latency p99. Alert on zero-result spike for popular prefixes.
Evolution
**v1 — SQL LIKE** — LIKE 'typ%'. Works to 10K queries.
**v2 — Redis trie** — Weekly rebuild. CDN. Debounce. 1B queries/day.
**v3 — Personalized** — Global trie + user boost re-rank. Hot injection. Multilingual shards.
Why it's hard to scale
Autocomplete looks read-heavy but the real trick is avoiding work: debounce cuts QPS 10×, CDN absorbs 90%, trie pre-computation eliminates subtree scans.
Key points
- **Trie with top-K per node** — Traverse to prefix node, return cached top-10 — no subtree walk.
- **Batch rebuild** — Do not update trie on every query. Weekly full rebuild + hot-term injection for breaking news.
- **Redis serving** — Key = prefix, value = suggestions array. Sub-ms lookup.
- **CDN for hot prefixes** — Top 10K prefixes cover ~90% of traffic.
- **Client debounce** — 100–200ms debounce cuts backend QPS ~10×.
- **Shard by prefix** — First character (or 2-gram) → independent Redis shard.
- **Content filter** — Blocklist inappropriate suggestions before storing in trie.
> Debounce, CDN, Redis trie — three free wins before any complex ML.
Tradeoffs
**Real-time trie update vs batch rebuild** — Batch is correct — slight staleness acceptable; real-time updates pollute the write path.
**Trie in memory vs Redis** — Redis shared across API fleet, easy blue-green trie deploy. In-memory fastest but harder to update.
**Top-K cached vs compute on query** — Pre-cached top-K avoids O(subtree) traversal — required at scale.
**Global vs personalized trie** — Per-user trie impossible at 1B users. Global trie + query-time re-rank boost from user history.
> "Batch-built trie with top-K at every node, served from Redis/CDN, debounced client queries."
Deep dives
#### Deep dive 1: Trie structure and top-K at every node
> [!CAUTION]
> **🔴 Weak** — scan all queries matching prefix on every keystroke
>
> [!WARNING]
> **🟡 Strong** — prefix tree where each node stores top-10 completions by global frequency. Query = O(prefix length) traversal + O(1) return
>
> [!TIP]
> **🟢 Staff+** — 1M unique prefixes × 10 suggestions × 50B ≈ 500 MB in Redis
#### Deep dive 2: Data pipeline — batch rebuild with hot-term injection
_Weekly MapReduce over query logs → frequency table → trie builder → shadow deploy → atomic flip. Breaking news: real-time detector flags queries with no trie match exceeding 1K/5min → inject temporary hot entry until next rebuild_
> [!CAUTION]
> **🔴 Weak** — Rebuild the full index nightly — no incremental updates.
>
> [!WARNING]
> **🟡 Strong** — Weekly MapReduce over query logs → frequency table → trie builder → shadow deploy → atomic flip. Breaking news: real-time detector flags queries with no trie match exceeding 1K/5min → inject temporary hot entry until next rebuild
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 3: Latency budget — debounce, CDN, sharding
_578K QPS raw keystrokes → 58K with debounce. CDN serves 90% from edge. Backend sees ~5.8K QPS for long tail. Shard Redis by first character; sub-shard hot prefixes like "th"_
> [!CAUTION]
> **🔴 Weak** — Serve every request from origin — CDN is optional.
>
> [!WARNING]
> **🟡 Strong** — 578K QPS raw keystrokes → 58K with debounce. CDN serves 90% from edge. Backend sees ~5.8K QPS for long tail. Shard Redis by first character; sub-shard hot prefixes like "th"
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 4: Personalization without per-user tries
_Fetch user boost vector from Redis (recent searches). Re-rank trie top-50 candidates in ~5ms. Decouple retrieval (trie) from ranking (lightweight model)_
> [!CAUTION]
> **🔴 Weak** — Build a per-user trie — one per user at scale.
>
> [!WARNING]
> **🟡 Strong** — Fetch user boost vector from Redis (recent searches). Re-rank trie top-50 candidates in ~5ms. Decouple retrieval (trie) from ranking (lightweight model)
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Two-pipeline script.
2. "Two components: offline trie builder and online query service."
3. "Offline: aggregate query logs daily, build trie with top-10 per prefix, load into Redis weekly with blue-green deploy."
4. "Online: debounce 100ms → Redis HGET prefix → return top-10 in <5ms."
5. "CDN caches top 10K prefixes — 90% of traffic never hits origin."
6. "Hot-term injection for viral queries between rebuilds."
7. "Shard trie by first character for horizontal scale."
Whiteboard
```
[Query logs] --> [MapReduce] --> [Trie Builder] --> [Redis shards]
^
[User keystroke] --> [Debounce] --> [API] --+--> [CDN hit?] --> return
| miss
v
[Redis prefix lookup]
```
Say offline build + online lookup. CDN and debounce are the scaling story.
Each ID is 64 bits: timestamp (ms) + worker_id (from ZooKeeper lease) + per-ms sequence. IDs are time-sortable and unique without central DB roundtrip per ID.
Problem
Generate unique, roughly time-ordered IDs at high throughput across thousands of servers.
Hard parts: coordination without per-ID DB calls, clock drift, and sequence overflow within one millisecond.
Failures
**Clock moves backward after NTP correction**
Risk duplicate IDs if not handled.
_Fix:_ Wait until clock >= last_timestamp. Refuse to generate and alert.
**ZooKeeper unavailable**
Cannot assign new worker_ids. New instances cannot start.
_Fix:_ Pre-allocated worker_id pools per instance. Renew lease in background.
**Sequence overflow within 1ms**
>4096 IDs in one ms on one worker.
_Fix:_ Wait next millisecond. Add second worker or widen sequence bits.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 1024 workers, 1M IDs/s cluster-wide |
| Read QPS | 1M IDs/s — trivial in-process |
| Write QPS | No DB writes per ID |
| Storage | Zero storage for ID generation itself |
| Cache math | ZK stores worker registry only — KB scale |
| Verdict | Throughput is not the problem. Correctness under clock drift is. |
Design decisions
**Snowflake vs UUID**
→ Snowflake for DB-primary keys
Sortable, index-friendly, high throughput.
_Revisit when:_ UUID v7 (time-ordered) as modern alternative.
**ZK vs DB for worker_id**
→ ZooKeeper/etcd
Fast lease, ephemeral nodes, designed for coordination.
_Revisit when:_ DB lease if ZK ops burden too high.
**Custom epoch start**
→ Custom epoch (Twitter 2010)
Extends effective timestamp range.
_Revisit when:_ Unix epoch if simplicity preferred.
Follow-up Q&A
**What if two workers get the same worker_id?**
ZK ephemeral nodes prevent duplicates. On conflict, one instance must exit and re-register.
**Can IDs leak information?**
Yes — timestamp and worker_id are embedded. Do not expose externally if enumeration is a concern.
**How do you migrate from auto-increment?**
Dual-write period: new rows get Snowflake, old rows keep int IDs. Application handles both types during migration.
**How do you handle leap seconds?**
NTP smears leap second. Monitor clock monotonicity. Some systems use logical clock instead of wall clock.
**Multi-region ID generation?**
Embed datacenter bits in worker_id range. Avoid cross-region ZK for latency.
**What about JavaScript Number precision?**
Snowflake exceeds JS safe integer — return IDs as strings in JSON APIs.
**How do you test uniqueness?**
Chaos: kill workers, force clock skew in test env, generate billions in parallel, verify no collisions.
**When is DB auto-increment fine?**
Single-region, <10K writes/s, no sortable ID requirement — Postgres sequence is simpler.
Evolution
**v1 — DB sequence** — SELECT nextval(). Simple. Bottleneck ~50K/s.
**v2 — Snowflake** — ZK worker leases. Per-process generation. Millions/s.
**v3 — Multi-DC** — DC-bit ranges. Monitoring for clock skew. String IDs in APIs.
Why it's hard to scale
ID generation is embarrassingly parallel once worker_ids are assigned. The coordination pain is clock sync and worker_id leasing, not throughput.
Key points
- **64-bit layout** — 41b timestamp | 10b worker_id | 12b sequence = 4096 IDs/ms per worker.
- **Worker ID lease** — ZooKeeper/etcd assigns worker_id. Ephemeral node — reclaimed on crash.
- **Clock synchronization** — NTP required. If clock moves backward: wait until caught up or fail loudly — never emit duplicate IDs.
- **Sequence per millisecond** — INCR sequence within same ms. Rollover → wait next ms.
- **No DB per ID** — Generation is in-process after worker_id assigned — millions/sec per node.
- **Sortable** — Time-ordered IDs useful for sharding and debugging.
- **Not cryptographic** — IDs are predictable — do not use as security tokens.
> Timestamp + worker + sequence. Mention clock backward handling — interviewers always ask.
Tradeoffs
**Snowflake vs DB auto-increment** — DB: bottleneck + single point. Snowflake: decentralized, sortable, no per-ID network call.
**Snowflake vs UUID v4** — UUID: random, not sortable, index fragmentation in B-trees. Snowflake: time-ordered, better DB index locality.
**Central counter vs Snowflake** — Central Redis INCR works but is SPOF and network per ID. Snowflake scales horizontally.
**48-bit vs 64-bit timestamp** — More timestamp bits = longer lifespan before overflow. Twitter Snowflake 41b ≈ 69 years.
> "Snowflake trades predictability for throughput and sortability — correct for internal IDs, not public tokens."
Deep dives
#### Deep dive 1: 64-bit layout and throughput math
_12-bit sequence = 4096 IDs/ms per worker. 10-bit worker = 1024 machines. Cluster theoretical max ≈ 4M IDs/ms_
> [!CAUTION]
> **🔴 Weak** — use UUID
>
> [!WARNING]
> **🟡 Strong** — explain bit allocation and why sortability helps DB indexing
>
> [!TIP]
> **🟢 Staff+** — Name the metric you'd alert on and when you'd revisit this design.
#### Deep dive 2: Clock drift and backward time
_NTP sync required. If current_ms < last_ms: wait or error. Never reuse timestamp+sequence combo_
> [!CAUTION]
> **🔴 Weak** — Use system clock on each machine — NTP is optional.
>
> [!WARNING]
> **🟡 Strong** — NTP sync required. If current_ms < last_ms: wait or error. Never reuse timestamp+sequence combo
>
> [!TIP]
> **🟢 Staff+** — leap seconds and VM migration can move clock backward — monitor and alert
#### Deep dive 3: Worker ID coordination
_ZooKeeper ephemeral sequential nodes assign worker_id. On crash, ID reclaimed after session timeout. Alternative: DB lease table with heartbeat — slower failover_
> [!CAUTION]
> **🔴 Weak** — Pick a random worker ID at process start.
>
> [!WARNING]
> **🟡 Strong** — ZooKeeper ephemeral sequential nodes assign worker_id. On crash, ID reclaimed after session timeout. Alternative: DB lease table with heartbeat — slower failover
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 4: Multi-datacenter IDs
_Per-DC worker_id ranges avoid cross-DC ZK dependency. Or dedicated ID service per region with DC bits in layout_
> [!CAUTION]
> **🔴 Weak** — UUID v4 everywhere — collisions are negligible.
>
> [!WARNING]
> **🟡 Strong** — Per-DC worker_id ranges avoid cross-DC ZK dependency. Or dedicated ID service per region with DC bits in layout
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Bit-layout script.
2. "Need unique, time-sortable IDs at millions/sec without DB per ID."
3. "Snowflake: 41b timestamp + 10b worker_id + 12b sequence."
4. "Worker_id from ZooKeeper ephemeral lease — reclaimed on crash."
5. "Sequence increments per millisecond; wait next ms on overflow."
6. "If clock goes backward, block generation — duplicates are worse than brief unavailability."
Whiteboard
```
[Service] --> [ID Generator lib]
|
worker_id from ZK lease
seq++ per millisecond
pack 64-bit ID
```
Draw three fields in the 64-bit ID. ZK assigns worker_id. Generation is local after lease.
Hotel reservation (Booking.com) Medium
Inventory lock · Saga · Idempotency
PostgreSQLRedisSELECT FOR UPDATESagaIdempotencyElasticsearch
Search reads Elasticsearch + Redis availability cache. Booking runs SELECT FOR UPDATE on inventory row, increments reserved count, creates PENDING reservation. Payment + email async via Kafka. Idempotency key prevents double-book on retry.
Problem
Search hotels, book rooms, prevent double-booking when two users grab the last room.
Correctness dominates — 167 bookings/sec is easy; double-book is catastrophic.
Failures
**Two users book last room**
Double-book without row lock.
_Fix:_ SELECT FOR UPDATE serializes concurrent bookings.
**Payment succeeds, DB write lost**
Charged customer, no reservation.
_Fix:_ Saga: create PENDING before charge. Compensation on failure.
**Stale Redis shows availability**
User sees room, book fails at DB.
_Fix:_ Cache advisory only. DB is source of truth.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 5K hotels, 100 rooms, 10K booking attempts/min peak |
| Read QPS | Search ~100 QPS peak — modest |
| Write QPS | 167 booking attempts/s peak — single MySQL primary fine |
| Storage | 182M inventory rows ≈ 36 GB |
| Cache math | Redis availability 1.5 GB |
| Verdict | Correctness engineering matters more than sharding. |
Design decisions
**Pessimistic vs optimistic**
→ SELECT FOR UPDATE
Flash sales create real contention — pessimistic wins.
_Revisit when:_ Optimistic for low-contention niche hotels.
**MySQL vs Cassandra for reservations**
→ MySQL ACID
Double-book on Cassandra is unacceptable.
_Revisit when:_ Cassandra for hotel metadata only.
**Reserve/pay ordering**
→ Reserve before charge
Never charge before durable reservation exists.
_Revisit when:_ Sync pay if PSP fast enough.
Follow-up Q&A
**How does the 10-minute hold work?**
PENDING_PAYMENT row reduces available count. Lock released after commit. Expiry job cancels stale holds.
**How do you search cheapest week in July?**
Precomputed daily min_price summary table. Flexible search queries summary, not raw inventory.
**How does Booking.com scale search?**
ES + Redis for search. MySQL only on booking path. Two-phase: approximate search, exact verification.
**How do you handle overbooking?**
Business allows reserved > total by buffer %. Configurable per hotel. Not a bug — revenue strategy.
**How do you handle cancellation?**
Decrement reserved, update reservation status, invalidate Redis cache for those dates.
**How do you prevent payment double-charge?**
Idempotency key on payment request tied to reservation_id.
**How do you shard at huge scale?**
Shard inventory by hotel_id. Bookings for one hotel single-shard ACID.
**How do you handle partial date ranges?**
Booking transaction locks one row per night in range. All must be available or entire booking fails.
Evolution
**v1 — Single DB** — FOR UPDATE only. Works for one hotel chain.
**v2 — Cache + saga** — Redis search cache. Async payment. Hold expiry.
**v3 — ES at scale** — Elasticsearch ranking. CDN hotel content. Multi-region.
Why it's hard to scale
Hotel reservation is a correctness problem. Contention spikes on popular dates; search volume dwarfs bookings but must not touch OLTP locks.
Key points
- **Inventory model** — room_inventory(hotel, type, date, total, reserved). Available = total - reserved.
- **Pessimistic lock** — SELECT FOR UPDATE on inventory row during booking transaction.
- **Idempotency key** — Client UUID — retry returns same reservation, no double charge.
- **Search vs book separation** — Search is approximate/fast (ES + Redis). Booking hits authoritative MySQL.
- **10-minute hold** — PENDING_PAYMENT reservation occupies slot. Background job expires holds.
- **Saga** — Reserve sync → pay async → confirm. Compensate (release inventory) if pay fails.
- **Cache advisory only** — Stale cache may show availability; DB lock prevents double-book.
> FOR UPDATE + idempotency + saga. Correctness trilogy.
Tradeoffs
**Pessimistic vs optimistic lock** — Pessimistic correct under flash-sale contention. Optimistic retries fail exactly when load is highest.
**Redis cache vs DB-only search** — 99% traffic is search — cache essential. Booking always validates DB.
**Sync vs async payment** — Reserve sync for instant feedback; pay async for PSP latency tolerance.
**Row inventory vs seat map** — Hotels: count-based rows. Airlines: per-seat map.
> "Search is approximate, booking is exact. SELECT FOR UPDATE is non-negotiable for inventory."
Deep dives
#### Deep dive 1: Double-booking prevention
_Transaction: SELECT reserved FROM inventory WHERE hotel=X AND date=Y FOR UPDATE. Check available>0. UPDATE reserved++. INSERT reservation. COMMIT. Second transaction blocks until first completes — sees updated count_
> [!CAUTION]
> **🔴 Weak** — UPDATE balance in SQL — no locking story.
>
> [!WARNING]
> **🟡 Strong** — Transaction: SELECT reserved FROM inventory WHERE hotel=X AND date=Y FOR UPDATE. Check available>0. UPDATE reserved++. INSERT reservation. COMMIT. Second transaction blocks until first completes — sees updated count
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 2: Hold window without long locks
_Lock held only for transaction duration (~50ms). PENDING_PAYMENT row holds inventory. Expiry job releases after 10 min. Payment at T+9:59 still valid if timestamp authoritative_
> [!CAUTION]
> **🔴 Weak** — Retry the charge on any timeout.
>
> [!WARNING]
> **🟡 Strong** — Lock held only for transaction duration (~50ms). PENDING_PAYMENT row holds inventory. Expiry job releases after 10 min. Payment at T+9:59 still valid if timestamp authoritative
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 3: Search at scale without touching OLTP
_Elasticsearch for hotel metadata/ranking. Redis for availability counts updated on booking. Search never acquires row locks_
> [!CAUTION]
> **🔴 Weak** — SELECT * WHERE column LIKE '%query%'.
>
> [!WARNING]
> **🟡 Strong** — Elasticsearch for hotel metadata/ranking. Redis for availability counts updated on booking. Search never acquires row locks
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 4: Saga compensation
_Pay fails → cancel reservation → decrement reserved. Idempotent compensation keyed by reservation_id_
> [!CAUTION]
> **🔴 Weak** — Oversimplify saga compensation — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Pay fails → cancel reservation → decrement reserved. Idempotent compensation keyed by reservation_id
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Correctness-first script.
2. "One challenge: prevent double-booking. Everything flows from that."
3. "Inventory: one row per hotel/room_type/date. Booking: SELECT FOR UPDATE, check available, increment reserved."
4. "Idempotency key on every book request."
5. "Search: ES + Redis cache — approximate. Booking: exact DB validation."
6. "Saga: reserve sync, pay + email async. Expire unpaid holds after 10 minutes."
Whiteboard
```
Search: User -> ES/Redis (approximate)
Book: User -> Booking Svc -> PG FOR UPDATE -> Kafka -> Payment -> Confirm
```
Two paths: fast approximate search, exact transactional booking.
Gaming leaderboard Medium
Redis ZSET · ZREVRANK · Top-K cache
Redis Sorted SetZINCRBYCassandraTop-KSeasonal
Match ends → ZINCRBY updates player score in Redis sorted set. Top-100 = ZREVRANGE 0 99. Individual rank = ZREVRANK in O(log N). Cassandra stores durable history to rebuild Redis after crash.
Problem
Real-time global leaderboard for millions of players: update scores instantly, return top-100 and any player rank fast.
Redis sorted set is purpose-built — DB RANK() scans are too slow at 100M players.
Failures
**Redis crash loses ZSET**
Ranks unknown until rebuild.
_Fix:_ AOF persistence. Rebuild from Cassandra.
**Concurrent score updates race**
Lost update if read-modify-write.
_Fix:_ ZINCRBY only — never GET then SET.
**1M simultaneous top-100 requests**
Redis read storm.
_Fix:_ Cache top-100 1s TTL.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 100M players, 1M active, 167K ZINCRBY/s, 100K top-100 reads/s |
| Read QPS | Top-100 cached → ~1 ZREVRANGE/s effective |
| Write QPS | 167K ZINCRBY/s — within Redis 1M ops/s |
| Storage | 100M × 40B ≈ 4 GB ZSET |
| Cache math | ZREVRANK 100K/s needs Redis cluster cores |
| Verdict | Individual rank reads scale harder than writes. |
Design decisions
**Seasonal reset**
→ New ZSET per season
Preserves history in Cassandra.
_Revisit when:_ Score reset in-place loses history.
**Friends leaderboard**
→ ZINTERSTORE on-demand
Precompute fan-out too expensive at 167K updates/s.
_Revisit when:_ Precompute for small friend graphs only.
**Anti-cheat**
→ Server validates score before ZINCRBY
Never trust client-reported final score.
_Revisit when:_ Replay validation for esports.
Follow-up Q&A
**How do daily/weekly/monthly boards coexist?**
Three ZSETs updated in one pipeline per match. Independent scores per period.
**How do you show players around me?**
ZREVRANK for R, ZREVRANGE R-5 R+5. Handle boundary at rank 0 and last rank.
**How do you scale to 1B players?**
40 GB still fits large Redis node. Write throughput driven by concurrent active users, not registered count.
**How do you paginate leaderboard?**
ZREVRANGE start stop WITHSCORES. Page size 100.
**How do you tie-break equal scores?**
Redis sorts by score then lexicographic member. Encode tie-breaker in member or use composite score (score * 1e6 + timestamp).
**How do you rebuild after Redis loss?**
Stream Cassandra scores, batch ZADD. Show 'rank updating' for individual ranks; serve cached top-100.
**How do you handle team leaderboards?**
Separate ZSET per team_id. Team score = sum of member ZINCRBY in pipeline.
**How do you prevent score inflation hacks?**
Server-side match validation. Anomaly detection on score velocity. Manual review queue.
Evolution
**v1 — SQL RANK()** — Minutes at 1M players.
**v2 — Redis ZSET** — Real-time rank. Cassandra backup.
**v3 — Multi-board** — Global/regional/friends/seasonal. Anti-cheat. Cached top-100.
Why it's hard to scale
Rank queries stay O(log N) as N grows. Real limits are write throughput during tournaments and top-100 read storms — both solved with ZINCRBY and short TTL cache.
Key points
- **Redis sorted set core** — ZADD/ZINCRBY/ZREVRANK/ZREVRANGE — all O(log N).
- **ZINCRBY not read-modify-write** — Atomic increment — no race on concurrent match results.
- **Cassandra durability** — Redis is serving layer; rebuild ZSET from Cassandra on failure.
- **Top-100 cache** — Cache ZREVRANGE result 1s TTL — tournament end storm.
- **Multiple boards** — Global, regional, friends, seasonal — separate ZSETs updated in pipeline.
- **Rank around me** — ZREVRANK then ZREVRANGE R-5 R+5.
- **Server-side validation** — Anti-cheat: validate score server-side before ZINCRBY.
> ZINCRBY, ZREVRANGE, ZREVRANK — three Redis commands, entire leaderboard.
Tradeoffs
**Redis ZSET vs SQL RANK()** — SQL RANK() is O(N log N). ZREVRANK is O(log N) — only correct answer for real-time rank.
**Real-time vs batch** — Players expect immediate rank change after match — real-time ZINCRBY required.
**Friends board: precompute vs on-demand** — ZINTERSTORE on-demand for small friend lists; precompute only if friends board is primary UI.
**Single ZSET vs sharded** — 100M players ≈ 4 GB — single node often enough. Shard when memory exceeds node capacity.
> "Redis sorted set is the right data structure — name ZINCRBY and ZREVRANK explicitly."
Deep dives
#### Deep dive 1: Why sorted set beats SQL
_At 100M players RANK() OVER scans entire table. ZREVRANK is ~27 comparisons. Top-100 is O(100) regardless of N_
> [!CAUTION]
> **🔴 Weak** — Oversimplify why sorted set beats sql — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — At 100M players RANK() OVER scans entire table. ZREVRANK is ~27 comparisons. Top-100 is O(100) regardless of N
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 2: Durability and rebuild
_Redis AOF + RDB. On total loss: batch ZADD from Cassandra — minutes for 100M players. Serve stale cached top-100 during rebuild_
> [!CAUTION]
> **🔴 Weak** — Oversimplify durability and rebuild — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Redis AOF + RDB. On total loss: batch ZADD from Cassandra — minutes for 100M players. Serve stale cached top-100 during rebuild
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 3: Write throughput and tournament storms
_167K ZINCRBY/s within Redis capacity. Top-100 cache with 1s TTL collapses read storm to 1 ZREVRANGE/s_
> [!CAUTION]
> **🔴 Weak** — Oversimplify write throughput and tournament storms — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — 167K ZINCRBY/s within Redis capacity. Top-100 cache with 1s TTL collapses read storm to 1 ZREVRANGE/s
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 4: Seasonal resets without downtime
_RENAME leaderboard:daily to leaderboard:daily:yesterday atomically at midnight. New empty set for new period_
> [!CAUTION]
> **🔴 Weak** — Oversimplify seasonal resets without downtime — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — RENAME leaderboard:daily to leaderboard:daily:yesterday atomically at midnight. New empty set for new period
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Data-structure-first script.
2. "Redis sorted set: ZINCRBY on score update, ZREVRANGE for top-100, ZREVRANK for my rank."
3. "ZINCRBY is atomic — concurrent match results safe."
4. "Cassandra stores history; Redis is serving layer."
5. "Cache top-100 one second during tournament ends."
6. "Multiple boards updated in one Redis pipeline."
Whiteboard
```
Match -> Score Service -> ZINCRBY leaderboard
|-> Cassandra (audit)
API -> ZREVRANGE top-100 (cached)
API -> ZREVRANK player_id
```
One diagram: write path ZINCRBY, read paths ZREVRANGE and ZREVRANK.
Topics split into partitions for parallelism. Consumer groups assign one consumer per partition. Replicas in ISR acknowledge writes for durability. Consumers commit offsets after processing.
Problem
Durable, high-throughput pub/sub log that decouples producers and consumers, scales horizontally, and replays history.
Hard parts: partition key design, consumer lag, and exactly-once vs at-least-once semantics.
Failures
**Hot partition key**
One partition overloaded; lag grows.
_Fix:_ Salt hot keys to sub-partitions or dedicated topic.
**Poison message infinite retry**
Consumer stuck, lag unbounded.
_Fix:_ DLQ after max retries.
**Rebalance storm**
Consumers pause frequently during deploy.
_Fix:_ Cooperative rebalancer. Static membership.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 1M events/s, 100 partitions, RF=3, 7-day retention |
| Read QPS | 1M consumer records/s with 100 consumers |
| Write QPS | 1M produce/s with proper partitioning |
| Storage | 1M/s × 1KB × 7d ≈ 600 TB — tiered storage |
| Cache math | Consumer lag metric is operational focus |
| Verdict | Partition count is capacity planning knob. |
Design decisions
**Partition count**
→ Start higher than needed — reducing is hard
Repartitioning changes key→partition mapping.
_Revisit when:_ 100–200 partitions per broker guideline.
**acks setting**
→ acks=all for critical data
acks=1 faster but can lose data on leader failure.
_Revisit when:_ acks=1 for metrics where loss acceptable.
**Kafka vs SQS**
→ Kafka for replay and high throughput
SQS simpler ops, no replay log.
_Revisit when:_ SQS for task queues without replay need.
Follow-up Q&A
**How do you preserve global ordering?**
Single partition — limits throughput. Usually order per key is enough.
**How do you add consumers without rebalance pain?**
Cooperative-sticky assignor. Incremental rebalance. Avoid frequent consumer restarts.
**How do you handle messages too large for Kafka?**
Store payload in S3, put pointer in Kafka message. Claim-check pattern.
**How do you migrate clusters?**
MirrorMaker 2 dual-write, switch consumers, drain old cluster.
**How do you achieve exactly-once?**
Idempotent producer + transactional writes + idempotent consumer. Higher latency.
**How do you prioritize topics?**
Separate clusters or dedicated broker pools per SLA tier.
**How do you compact topics?**
log.compaction for changelog topics — keeps latest per key.
**How do you debug consumer lag?**
Check slow processing, GC pauses, insufficient partitions, hot keys.
Evolution
**v1 — Single broker** — Dev only. No HA.
**v2 — Production cluster** — RF=3, ISR, monitoring, DLQ.
**v3 — Multi-DC** — MirrorMaker, tiered storage, exactly-once where needed.
Why it's hard to scale
Kafka scales by adding partitions and brokers. Limits: partition count planning, consumer lag under slow workers, hot keys.
Key points
- **Partition by key** — Same key → same partition preserves ordering per entity.
- **Consumer group scaling** — Max parallelism = partition count. More consumers than partitions sit idle.
- **Replication factor 3** — Leader + followers. ISR replicas must ack before commit (configurable).
- **Retention** — Log retained days/weeks — consumers can replay or catch up.
- **At-least-once default** — Commit offset after process. Idempotent consumers handle duplicates.
- **Dead letter topic** — Poison messages after N failures — do not block partition.
- **Monitor consumer lag** — Lag = high-priority alert. Autoscale consumers on lag.
> Partition key, consumer group, offset commit — Kafka trilogy.
Tradeoffs
**Kafka vs RabbitMQ** — Kafka: high-throughput log, replay, retention. RabbitMQ: task queues, routing, lower latency per message.
**At-least-once vs exactly-once** — Exactly-once needs transactions/idempotent producer — higher latency. At-least-once + idempotent consumer is default.
**More partitions vs bigger messages** — Partitions scale throughput; large messages need compression or external blob store.
**Push vs pull consumers** — Kafka consumers pull — control their own pace, natural backpressure.
> "Partition for ordering, consumer group for scale, offset commit for recovery."
Deep dives
#### Deep dive 1: Partition key design
_Wrong key (constant) → one hot partition. Right key (user_id, order_id) spreads load and preserves per-entity order_
> [!CAUTION]
> **🔴 Weak** — Oversimplify partition key design — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Wrong key (constant) → one hot partition. Right key (user_id, order_id) spreads load and preserves per-entity order
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 2: Consumer groups and rebalancing
_Adding consumer triggers rebalance — brief pause. Use cooperative sticky assignor to minimize disruption. Max consumers = partitions_
> [!CAUTION]
> **🔴 Weak** — Oversimplify consumer groups and rebalancing — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Adding consumer triggers rebalance — brief pause. Use cooperative sticky assignor to minimize disruption. Max consumers = partitions
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 3: Replication and ISR
_min.insync.replicas=2 with acks=all prevents data loss on broker failure. Unclean leader election trades availability for loss risk — avoid for financial topics_
> [!CAUTION]
> **🔴 Weak** — UUID v4 everywhere — collisions are negligible.
>
> [!WARNING]
> **🟡 Strong** — min.insync.replicas=2 with acks=all prevents data loss on broker failure. Unclean leader election trades availability for loss risk — avoid for financial topics
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 4: Handling poison pills
_After 3 failures route to DLQ. Skip bad message so partition progresses. Alert on DLQ depth_
> [!CAUTION]
> **🔴 Weak** — Oversimplify handling poison pills — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — After 3 failures route to DLQ. Skip bad message so partition progresses. Alert on DLQ depth
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Log-oriented script.
2. "Kafka is a durable commit log, not a traditional queue."
3. "Producers write to topic partitions. Partition key chooses partition."
4. "Consumer group: each partition consumed by one consumer in group."
5. "Commit offset after successful processing — at-least-once."
6. "Monitor consumer lag. DLQ for poison messages."
Whiteboard
```
Producers -> Kafka Topic (P0..Pn) -> Consumer Group -> Workers
\-> replicas in ISR
```
Draw partitions and consumer group. Mention ISR for durability.
Consistent hashing maps keys to nodes; adding a node remaps only K/n keys. Writes go to N replicas; reads use quorum (R+W>N). Vector clocks detect concurrent writes for conflict resolution.
Problem
Build Dynamo-style highly available KV store: partition data across nodes, replicate for durability, remain available during partitions.
Tradeoff: availability + partition tolerance vs strong consistency.
Failures
**Hot key overloads one node**
Single key QPS exceeds node capacity.
_Fix:_ Client-side cache. Key splitting (hot_key:1, hot_key:2) merge on read.
**Node failure during write**
Write unavailable if strict quorum.
_Fix:_ Sloppy quorum + hinted handoff.
**Replica divergence**
Silent data drift between nodes.
_Fix:_ Merkle tree anti-entropy background repair.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 1B keys, 10K QPS reads, 5K QPS writes, N=3 |
| Read QPS | 10K reads/s with R=2 quorum = 20K internal reads/s |
| Write QPS | 5K×W=2 = 10K internal writes/s |
| Storage | 1B × 1KB = 1 TB total — 333 GB per node at RF=3 |
| Cache math | Hot key cache on client reduces 80% hot traffic |
| Verdict | Hot keys and quorum math dominate design discussion. |
Design decisions
**W and R tuning**
→ W=2,R=2,N=3 for balanced
Higher W+R = stronger consistency, higher latency.
_Revisit when:_ W=1,R=1 for cache-like tolerance of staleness.
**Leaderless vs Raft per shard**
→ Leaderless for availability
Dynamo/Cassandra model. Raft per shard (TiKV) gives stronger consistency at cost.
_Revisit when:_ Raft if financial/adjacent strong consistency needed.
**Delete handling**
→ Tombstone with vector clock
Deletes are writes — propagate like puts.
_Revisit when:_ TTL automatic expiry for ephemeral keys.
Follow-up Q&A
**How is this different from Redis?**
Redis is in-memory, often single-primary per shard, lower latency. Dynamo KV is disk-backed, leaderless, higher availability during partitions.
**How do you add a node?**
Join ring, steal vnodes from neighbors, stream data, go live. Minimal key remapping.
**How do you handle network partition?**
AP choice: both sides accept writes. Vector clocks detect conflicts on heal.
**How do you cap stale reads?**
Increase R and W. R+W>N guarantees overlap with latest write.
**How do you implement CAS?**
Read vector clock, put only if clock matches expected — lightweight transaction.
**How do you monitor health?**
Gossip membership, per-node latency, repair lag, sibling rate metric.
**How do you backup?**
Incremental snapshots per node + cross-region replication.
**When not to build this?**
If Postgres/Redis/DynamoDB managed service fits — build only when hyperscale or custom CAP point needed.
Evolution
**v1 — Single node** — Redis/Postgres. No partition tolerance.
**v2 — Sharded replicas** — Consistent hash, RF=3, quorum reads/writes.
**v3 — Production Dynamo** — Hinted handoff, Merkle repair, tunable W/R, monitoring.
Why it's hard to scale
KV store scales by adding nodes to the ring. Pain points: hot keys, quorum latency, conflict resolution complexity.
Key points
- **Consistent hashing + vnodes** — Even load distribution; minimal remapping on node add/remove.
- **Replication factor N=3** — Write to N nodes. Tolerate N-1 failures with proper quorum.
- **Quorum reads/writes** — W writes, R reads, R+W>N gives consistency guarantee.
- **Vector clocks** — Detect concurrent updates; client or system resolves siblings.
- **Sloppy quorum + hinted handoff** — Write to W healthy nodes even if primary down; hand off when node returns.
- **Anti-entropy** — Merkle tree sync repairs divergence between replicas.
- **CAP choice** — AP system — available during partition, eventual consistency.
> Consistent hash, quorum, vector clocks — Dynamo paper vocabulary.
Tradeoffs
**Strong consistency vs eventual** — Quorum with R+W>N approaches strong for single-key ops. Full strong needs consensus (Paxos/Raft) per write — slower.
**Leader-based vs leaderless** — Leaderless (Dynamo/Cassandra) better availability. Leader (Redis primary) simpler consistency.
**Modulo vs consistent hash** — Modulo remaps all keys on resize — unacceptable at scale.
**LWW vs vector clocks** — Last-write-wins loses data on concurrent edits. Vector clocks preserve conflict history.
> "AP KV with quorum tuning — expose consistency/latency tradeoff via W and R."
Deep dives
#### Deep dive 1: Consistent hashing and virtual nodes
_Without vnodes, ring imbalance 3×. 150 vnodes per physical node smooths distribution. Node add/remove remaps only adjacent key ranges_
> [!CAUTION]
> **🔴 Weak** — Oversimplify consistent hashing and virtual nodes — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Without vnodes, ring imbalance 3×. 150 vnodes per physical node smooths distribution. Node add/remove remaps only adjacent key ranges
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 2: Quorum math
_N=3, W=2, R=2 → R+W>N guarantees read sees latest write. W=1, R=1 fastest but stale reads possible. Tune per use case_
> [!CAUTION]
> **🔴 Weak** — Oversimplify quorum math — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — N=3, W=2, R=2 → R+W>N guarantees read sees latest write. W=1, R=1 fastest but stale reads possible. Tune per use case
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 3: Failure handling — hinted handoff and Merkle trees
_Node down: write to alternative node with hint. On recovery, hand off data. Background Merkle tree comparison finds drift_
> [!CAUTION]
> **🔴 Weak** — Oversimplify failure handling — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Node down: write to alternative node with hint. On recovery, hand off data. Background Merkle tree comparison finds drift
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 4: Conflict resolution
_Concurrent puts create sibling versions. Client reads all siblings, merges (e.g., cart union), writes resolved version_
> [!CAUTION]
> **🔴 Weak** — Oversimplify conflict resolution — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Concurrent puts create sibling versions. Client reads all siblings, merges (e.g., cart union), writes resolved version
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Dynamo vocabulary script.
2. "Partition with consistent hashing. Replicate N=3."
3. "Quorum: W writes, R reads, R+W>N for consistency."
4. "Vector clocks detect concurrent writes."
5. "Hinted handoff maintains availability during node failure."
6. "Merkle trees for anti-entropy repair."
Whiteboard
```
Client -> Coordinator -> hash ring -> Replica nodes (N=3)
W writes, R reads, gossip membership
```
Ring diagram + quorum numbers on whiteboard.
Nearby friends Hard
Geohash grid · Redis GEO · Location fan-out
Redis GEOGeohashWebSocketQuadtreeLocation updates
Users report GPS every 30s → geohash cell → Redis GEOADD in per-cell sorted set. Finding nearby friends = query same + adjacent cells. Notify matches via WebSocket. Location is ephemeral — not written to PostgreSQL.
Problem
Show users which friends are physically nearby in real time.
Hard parts: high-frequency location updates, efficient geo queries, and privacy/battery constraints.
Failures
**User on cell boundary**
Missed nearby friends if only querying one cell.
_Fix:_ Always query cell + 8 neighbors.
**Redis memory exhaustion**
Millions of live locations.
_Fix:_ TTL on keys. Only store active users last 5 min.
**Push notification spam**
Notify every location tick.
_Fix:_ State machine: notify only on enter/exit radius.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 50M DAU, 2M concurrent sharing location, update/30s |
| Read QPS | Nearby queries: 2M/60s ≈ 33K/s |
| Write QPS | 2M/30s ≈ 67K GEOADD/s |
| Storage | 2M × 100B ≈ 200 MB Redis |
| Cache math | Friend list cache per user in Redis |
| Verdict | Redis handles write rate; query uses cell intersection. |
Design decisions
**Redis GEO vs custom geohash**
→ Redis GEO (geohash under hood)
Built-in GEOADD/GEORADIUS — battle-tested.
_Revisit when:_ Custom quadtree for uneven density maps.
**Update frequency**
→ Adaptive 30s–5min
Battery and write volume scale with frequency.
_Revisit when:_ Fixed 30s if product requires live map.
**Friend graph lookup**
→ Cache friend IDs in Redis per user
Avoid PG join on every geo query.
_Revisit when:_ PG query if friend list small and rare.
Follow-up Q&A
**How do you handle users without GPS?**
Fall back to coarse IP geolocation — mark precision as LOW in UI.
**How do you support 'nearby strangers' mode?**
Skip friend intersection — query all users in cell. Stronger privacy fuzzing required.
**How do you scale to multiple cities?**
Shard Redis by geohash prefix region. Users query local shard.
**How do you prevent stalking?**
Mutual opt-in, precision limits, sharing session timeout, block list.
**How do you test geo correctness?**
Unit test haversine + cell neighbor coverage. Simulation with known coordinates.
**How do you handle indoor GPS drift?**
Kalman filter smooth positions. Minimum movement threshold before updating cell.
**How do you show 'last seen' vs live?**
Separate last_seen timestamp in PG; live map uses Redis TTL presence only.
**How does this differ from Yelp geo search?**
Yelp searches businesses (static POIs). Nearby friends tracks moving users with ephemeral coordinates.
Evolution
**v1 — Poll + PG** — Store lat/long in PG. Too slow for live.
**v2 — Redis GEO** — Ephemeral locations. Geohash queries. Friend intersection.
**v3 — Production** — Adaptive updates, privacy modes, enter/exit push FSM, regional sharding.
Why it's hard to scale
High-frequency ephemeral writes and geo indexing are the pain points — not friend graph size.
Key points
- **Ephemeral location in Redis** — 2M users × update/30s = 67K writes/s — Redis yes, PostgreSQL no.
- **Geohash cells** — Map lat/long to cell. Query cell + 8 neighbors covers edge cases.
- **Friend graph in PG** — Friendships durable in PostgreSQL. Intersect geo results with friend list.
- **Push on proximity** — WebSocket notify when friend enters radius — not poll.
- **Privacy controls** — Ghost mode, precision reduction, sharing window.
- **Battery** — Adaptive update interval: stationary → 5 min, moving → 30s.
- **Stale location TTL** — EXPIRE location keys 5 min — no ghost users on map.
> Redis for location, PG for friendships, geohash for query.
Tradeoffs
**Redis GEO vs PostGIS** — Redis: in-memory, ephemeral, fast. PostGIS: durable, complex queries, slower — wrong for live location stream.
**Geohash vs quadtree** — Geohash simpler with Redis. Quadtree better for variable density — more complex.
**Push vs poll for updates** — WebSocket push for real-time map. Poll drains battery.
**Exact GPS vs fuzzed** — Fuzz location to ~100m for privacy unless user opts in to precise.
> "Location is ephemeral in Redis; friendships durable in PG; geohash makes neighbor search O(cells) not O(users)."
Deep dives
#### Deep dive 1: Write path at scale
_67K GEOADD/s is Redis-comfortable. Never persist every fix to PG. TTL keys expire stale users_
> [!CAUTION]
> **🔴 Weak** — Oversimplify write path at scale — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — 67K GEOADD/s is Redis-comfortable. Never persist every fix to PG. TTL keys expire stale users
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 2: Query algorithm
_Compute user cell. Fetch users in cell + 8 neighbors. Filter by haversine distance < R. Intersect with friend IDs from PG/cache_
> [!CAUTION]
> **🔴 Weak** — Oversimplify query algorithm — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Compute user cell. Fetch users in cell + 8 neighbors. Filter by haversine distance < R. Intersect with friend IDs from PG/cache
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 3: Privacy and precision
_Reduce precision to 5-char geohash (~5km) by default. Ghost mode deletes Redis entry immediately_
> [!CAUTION]
> **🔴 Weak** — Oversimplify privacy and precision — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Reduce precision to 5-char geohash (~5km) by default. Ghost mode deletes Redis entry immediately
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 4: Notification dedup
_Do not push every 30s if friend still nearby. State machine: ENTERED_NEARBY → INSIDE → EXITED. Push only on transitions_
> [!CAUTION]
> **🔴 Weak** — Retry until delivery succeeds — duplicates are rare.
>
> [!WARNING]
> **🟡 Strong** — Do not push every 30s if friend still nearby. State machine: ENTERED_NEARBY → INSIDE → EXITED. Push only on transitions
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Geo + graph script.
2. "Location updates high frequency — Redis only, TTL 5 min."
3. "Geohash cell + neighbors for nearby query."
4. "Intersect geo candidates with friend list from PG."
5. "WebSocket push on enter/exit proximity, not continuous poll."
6. "Privacy: fuzz precision, ghost mode, adaptive update rate."
Whiteboard
```
GPS -> Location Svc -> Redis GEO (per geohash cell)
Friend list <- PostgreSQL
Matcher -> intersect -> WebSocket push
```
Two stores: ephemeral geo in Redis, social graph in PG.
Google Maps Hard
Tile CDN · Geospatial index · Routing graph
S3/GCSCDNQuadtreePostGISDijkstraTile cache
Map rendering is CDN-served tiles (z/x/y) stored in S3 — immutable, cacheable for years. Place search uses Elasticsearch with geo filters. Routing runs on a preprocessed road graph (not live OSM queries) with contraction hierarchies for sub-second paths.
Problem
Build a global maps platform: render maps fast worldwide, search places, and compute driving directions at scale.
Hard parts: petabytes of tiles, sub-100ms pan/zoom, and routing on a graph with hundreds of millions of edges.
Failures
**CDN miss storm on new region launch**
Origin tile service overwhelmed.
_Fix:_ Pre-warm CDN. Rate limit origin. Autoscale tile generators.
**Stale traffic overlay**
Users routed into closed roads.
_Fix:_ Separate traffic freshness SLA. Fallback to historical speeds.
**POI index drift**
New businesses missing from search.
_Fix:_ CDC from merchant DB + nightly full rebuild.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 500M DAU, 50 tile requests/session, 10M routing requests/day |
| Read QPS | Tiles: 500M×50/86400 ≈ 290K/s — CDN absorbs |
| Write QPS | Routing: 10M/86400 ≈ 115/s compute |
| Storage | Zoom 0–18 global pyramid ≈ petabytes — store regional hot sets |
| Cache math | CDN cache hot z/x/y prefixes |
| Verdict | CDN is the scaling lever for tiles; routing needs graph sharding. |
Design decisions
**Raster vs vector tiles**
→ Raster for interview default
CDN-friendly, simple. Vector if interviewer asks about dynamic styling.
_Revisit when:_ Vector for offline mobile maps.
**Routing algorithm**
→ Contraction hierarchies
Sub-second on continental graphs after preprocessing.
_Revisit when:_ A* on small metro graphs only.
**Traffic freshness**
→ Separate dynamic layer
Base tiles stay immutable; traffic updates frequently.
_Revisit when:_ Bake traffic into tiles only for replay/historical.
Follow-up Q&A
**How do you generate tiles at scale?**
Batch MapReduce over planet data. Parallel workers per z/x/y batch. Store to S3. Long tail on-demand generation with cache.
**How do you handle map updates (new roads)?**
Versioned tile sets. Client requests v=2026-06. Gradual CDN rollouts per region.
**How do you rank search results?**
BM25 text score × exp(-distance/λ) × log(popularity). Personalization optional.
**How do you support offline maps?**
Bundle vector tiles + local routing subgraph on device. Sync deltas weekly.
**How do you reduce routing latency globally?**
Regional routing shards. Cross-border: coarse inter-region graph first.
**How do you detect map vandalism?**
Human review queue + automated anomaly detection on edits.
**What metrics matter?**
CDN hit ratio, tile origin QPS, routing p99, search zero-result rate.
**How do you test routing correctness?**
Golden paths vs known benchmarks. A/B on ETA accuracy vs ground truth.
Evolution
**v1 — Static tiles** — Prebuilt tiles + CDN. No live routing.
**v2 — Search + routing** — ES POI index. Regional routing graphs.
**v3 — Global** — Traffic overlay, CH routing, multi-region CDN, map edit pipeline.
Why it's hard to scale
Petabyte-scale immutable tiles and global CDN hit ratio dominate — routing is compute-heavy but smaller QPS.
Key points
- **Tile pyramid** — Zoom level z has 4^z tiles globally. Pre-generate popular regions; on-demand for long tail.
- **CDN-first** — Tiles are immutable — Cache-Control: max-age=31536000. CDN handles 99% of map traffic.
- **Geospatial search** — POIs indexed in ES with geo_point. Query: text match + geo_distance filter + popularity boost.
- **Routing graph** — Preprocessed road network offline. Online: bidirectional Dijkstra or contraction hierarchies.
- **Live traffic** — Traffic overlay is dynamic — separate tile layer or client-side vector update, not baked into base tiles.
- **Personalization out of scope** — Unless asked: saved places, ads, Street View capture pipeline.
> Tiles on CDN, POIs in search index, routing on preprocessed graph.
Tradeoffs
**Raster tiles vs vector tiles** — Raster: simpler CDN caching. Vector: smaller payloads, client-side styling — more client CPU.
**On-demand routing vs precomputed** — Precompute hub labels for fast queries. On-demand Dijkstra only for local refinement.
**PostGIS vs Elasticsearch for POI** — ES wins for text+geo hybrid search at scale. PostGIS for complex polygon queries.
> "Immutable tiles on CDN, search index for POIs, offline graph preprocessing for routing."
Deep dives
#### Deep dive 1: Tile storage and CDN
> [!CAUTION]
> **🔴 Weak** — generate tiles per request
>
> [!WARNING]
> **🟡 Strong** — pre-render pyramid, store in S3, serve via CDN. Staff+: invalidation only for traffic/incident overlays
>
> [!TIP]
> **🟢 Staff+** — Name the metric you'd alert on and when you'd revisit this design.
#### Deep dive 2: POI search ranking
_Text relevance × distance decay × popularity. Geo filter first to shrink candidate set_
> [!CAUTION]
> **🔴 Weak** — SELECT * WHERE column LIKE '%query%'.
>
> [!WARNING]
> **🟡 Strong** — Text relevance × distance decay × popularity. Geo filter first to shrink candidate set
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 3: Routing at scale
_Graph partitioned by region. Highway hierarchy: coarse graph for long distances, refine locally_
> [!CAUTION]
> **🔴 Weak** — Oversimplify routing at scale — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Graph partitioned by region. Highway hierarchy: coarse graph for long distances, refine locally
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 4: Fresh traffic data
_Probe GPS stream → aggregate speeds per road segment → publish traffic layer every 2–5 min_
> [!CAUTION]
> **🔴 Weak** — Oversimplify fresh traffic data — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Probe GPS stream → aggregate speeds per road segment → publish traffic layer every 2–5 min
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Map platform script.
2. "Three paths: tile rendering (read-heavy CDN), place search (ES + geo), routing (preprocessed graph)."
3. "Tiles: z/x/y in object storage, immutable, CDN cached forever."
4. "Search: Elasticsearch geo_point + text, rank by distance and popularity."
5. "Routing: contraction hierarchies on offline graph — not live OSM queries per request."
Whiteboard
```
Client -> CDN -> S3 tiles (base map)
Client -> API -> ES (POI search)
Client -> Routing svc -> Graph shards (CH / hub labels)
```
Separate read paths for tiles, search, and routing.
Distributed email (Gmail) Hard
Ingestion · Storage sharding · Search index
SMTPCassandraHDFSMapReduceElasticsearchSpam ML
Inbound SMTP mail is parsed, scanned for spam, and written to a user shard (Cassandra). Large attachments go to blob storage. Elasticsearch indexes subject/body asynchronously for search. Reads hit the user shard directly — email is write-once, read-many per mailbox.
Problem
Design webmail at Gmail scale: receive, store, search, and send billions of emails with strong per-user consistency.
Hard parts: storage per user, full-text search, and reliable SMTP delivery.
Failures
**Hot user shard (celebrity inbox)**
Millions of fan emails to one user.
_Fix:_ Rate limit per sender. Separate fan-mail bucket. Async fan-out to readers.
**Search index lag**
New mail not findable for minutes.
_Fix:_ Monitor indexer lag. Priority queue for recent mail.
**Attachment virus**
Malware stored in blob system.
_Fix:_ Scan before blob write. Block executable MIME types.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 1B users, 100 emails/user/day avg, 50KB avg with attachments |
| Read QPS | Inbox read: 1B×20/86400 ≈ 230K/s |
| Write QPS | Ingest: 1B×100/86400 ≈ 1.16M/s |
| Storage | 100B emails/day × 50KB ≈ 4.5 PB/day raw — tiered storage + dedup |
| Cache math | Hot inbox cache per user in Redis |
| Verdict | Write sharding by user_id is the core decision. |
Design decisions
**Cassandra vs HDFS for mail bodies**
→ Cassandra hot + HDFS cold archive
Recent mail low-latency. Archive cheap on HDFS/Glacier.
_Revisit when:_ All Cassandra if simplifying interview.
**Global vs per-user search index**
→ Global ES with user_id filter
Simpler ops. Per-user index only for enterprise vaults.
_Revisit when:_ Per-user index at extreme privacy requirements.
**Strong consistency for inbox**
→ Quorum reads on user partition
User expects read-your-writes after send.
_Revisit when:_ Eventual for search index only.
Follow-up Q&A
**How do you handle duplicate delivery (SMTP retry)?**
Message-ID dedup at ingestion. Store idempotency key 7 days.
**How do you implement labels/folders?**
Secondary index table: user_id + label → message_ids. Or bitmap per label.
**How do you support full mailbox search?**
ES query with user_id filter + highlighting. Fallback to metadata scan if index lag.
**How do you migrate a user between shards?**
Dual-write period. Background copy. Flip read route. Delete old after verify.
**How do you handle court-ordered retention?**
Legal hold flag bypasses user delete. Separate retention policy store.
**How do you scale SMTP ingress?**
Stateless SMTP proxies → Kafka → storage writers. Horizontal scale on proxies.
**How do you prevent outbound spam?**
Per-user send quotas. Reputation score. Delay suspicious bulk sends.
**How do you measure deliverability?**
Bounce rate, complaint rate, time-to-inbox, indexer lag.
Evolution
**v1 — Single DB** — Postgres per mail. Works to millions of messages.
**v2 — Sharded** — Cassandra user shards. Blob attachments. Async search.
**v3 — Gmail scale** — Tiered storage, ML spam, global ES, SMTP fleet, legal hold.
Why it's hard to scale
Billions of messages/day with large attachments — sharding and blob offload are mandatory.
Key points
- **Shard by user_id** — All of a user's mail on one shard — simplifies inbox listing and ACID per mailbox.
- **Blob for attachments** — Message metadata in Cassandra; attachment bytes in S3/HDFS.
- **Async search index** — Kafka mail event → ES indexer. Search slightly behind inbox — acceptable.
- **Spam at ingress** — Score before storage. Quarantine bucket for suspicious mail.
- **SMTP outbound queue** — Retry with exponential backoff. DKIM/SPF signing per domain.
- **Deletion/tombstones** — Soft delete + async purge from index and blob store.
> User-sharded storage, blob attachments, async search index.
Tradeoffs
**SQL vs Cassandra per user shard** — Cassandra: write-heavy, tunable consistency, horizontal scale. SQL: simpler but harder at Gmail scale.
**Sync vs async search index** — Async: faster ingest. Sync: instant search — costly at write volume.
**Push vs poll for new mail** — IMAP IDLE / WebSocket push for new mail notifications.
> "Shard mail by user, blob store attachments, async ES index, spam filter at SMTP ingress."
Deep dives
#### Deep dive 1: Storage model
> [!CAUTION]
> **🔴 Weak** — one row per email in SQL
>
> [!WARNING]
> **🟡 Strong** — Cassandra partition key = user_id, cluster key = timestamp+id. Staff+: separate hot (inbox) and cold (archive) tiers
>
> [!TIP]
> **🟢 Staff+** — Name the metric you'd alert on and when you'd revisit this design.
#### Deep dive 2: Search
_Inverted index per user or global with user_id filter. Reindex pipeline from mail log for recovery_
> [!CAUTION]
> **🔴 Weak** — Rebuild the full index nightly — no incremental updates.
>
> [!WARNING]
> **🟡 Strong** — Inverted index per user or global with user_id filter. Reindex pipeline from mail log for recovery
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 3: SMTP reliability
_Outbound queue in Kafka. Multiple MX retries. Bounce handling updates recipient reputation_
> [!CAUTION]
> **🔴 Weak** — Oversimplify smtp reliability — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Outbound queue in Kafka. Multiple MX retries. Bounce handling updates recipient reputation
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 4: Spam/abuse
_Feature extraction at edge. ML model ensemble. User feedback loop for false positives_
> [!CAUTION]
> **🔴 Weak** — Query the database on every feed request.
>
> [!WARNING]
> **🟡 Strong** — Feature extraction at edge. ML model ensemble. User feedback loop for false positives
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Email platform script.
2. "Inbound SMTP → parse → spam scan → user shard. Attachments to blob store."
3. "Search: async indexer from mail events — inbox read path does not wait on ES."
4. "Outbound: queue + retry + DKIM. Per-user send rate limits."
Whiteboard
```
SMTP -> Ingest -> Spam -> Cassandra (user shard) -> API -> Client
-> Blob (attachments)
-> Kafka -> ES indexer
```
User shard is source of truth; search is derived.
Clients call the API gateway. The metadata service maps bucket/key → data node locations via consistent hashing. Object bytes live on data nodes with replication (and erasure coding for cold tier). Large uploads use multipart with coordinator assembly.
Problem
Design S3-like object storage: PUT/GET/DELETE objects, 11-nines durability, unlimited scale, presigned URLs.
Hard parts: metadata scale, rebalancing on node failure, and large object uploads.
Failures
**Node failure during PUT**
Incomplete object visible.
_Fix:_ Write to temp key. Commit metadata only after all replicas ACK.
**Hot key (viral object)**
One object saturates single node.
_Fix:_ CDN in front. Replication already helps reads. Split hot objects across cache.
**Ring imbalance**
Some nodes 2× fuller than others.
_Fix:_ Virtual nodes. Background rebalancer moves ranges.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 1T objects, 100K PUT/s peak, 1M GET/s peak, 1MB avg object |
| Read QPS | 1M GET/s — CDN serves 90% |
| Write QPS | 100K PUT/s × 3 replicas = 300K disk writes/s cluster-wide |
| Storage | 1T × 1MB = 1 EB logical — EC reduces physical |
| Cache math | Metadata: 1T keys × 500B = 500 TB metadata — shard buckets |
| Verdict | Metadata sharding and CDN are critical at this scale. |
Design decisions
**Replication vs erasure coding**
→ 3x replication hot, EC cold
Interviewers accept tiered durability.
_Revisit when:_ All EC if cost is the focus.
**Metadata store**
→ Cassandra partitioned by bucket
Horizontal scale for object index.
_Revisit when:_ FoundationDB/etcd for smaller scale.
**Strong consistency**
→ Quorum writes + leader for metadata
Read-after-write on new keys matters for clients.
_Revisit when:_ Eventual for cross-region async replication.
Follow-up Q&A
**How do presigned URLs work?**
HMAC(bucket, key, expiry, secret). Gateway validates signature before allowing PUT/GET.
**How do you delete objects at scale?**
Tombstone metadata. Async garbage collect bytes when ref count zero.
**How do you implement versioning?**
Version ID in metadata. DELETE marker for latest. List versions API.
**How do you handle concurrent writers?**
If-none-match / version checks. Last writer wins or reject conflict.
**How do you migrate a bucket between shards?**
Background copy with dual metadata. Cutover per key prefix.
**How do you monitor durability?**
Bit rot scrub. Replica lag. Missing replica alerts.
**How do you support cross-region replication?**
Async replicate bytes + metadata. CRR queue per object.
**How does this relate to Dropbox?**
Dropbox adds sync metadata + chunk dedup on top of object storage primitives.
Evolution
**v1 — Single machine** — Disk + SQLite metadata.
**v2 — Hash ring** — Data nodes + metadata service + replication.
**v3 — S3-class** — Multipart, EC tiers, CDN, cross-region, lifecycle policies.
Why it's hard to scale
Petabyte payloads and metadata billions of keys — hashing and tiering dominate.
Key points
- **Metadata vs data separation** — Small metadata in SQL/Cassandra. Payload on commodity disks.
- **Consistent hashing** — Ring with virtual nodes. Add/remove nodes with minimal reshuffle.
- **Replication** — 3 replicas across racks/AZs. Quorum write before ACK.
- **Erasure coding** — Cold/archive tier: 10+4 EC reduces storage cost vs 3x replication.
- **Multipart upload** — Split >100MB into parts. Parallel upload. Commit manifest on complete.
- **Presigned URLs** — HMAC token lets client upload/download without proxying bytes through API.
> Metadata ring mapping, replicated data nodes, multipart for large objects.
Tradeoffs
**3x replication vs erasure coding** — Replication: faster reads, hotter tier. EC: cheaper for cold data.
**Strong listing consistency vs eventual** — S3 now strong for read-after-write on new objects. Listing can lag slightly.
**Central metadata vs per-bucket partition** — Partition metadata by bucket hash for scale.
> "Consistent hash for placement, metadata service for lookup, replication for durability."
Deep dives
#### Deep dive 1: Consistent hashing and rebalancing
_Virtual nodes smooth load. On node add: steal ranges. On failure: replicate to successor_
> [!CAUTION]
> **🔴 Weak** — Oversimplify consistent hashing and rebalancing — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Virtual nodes smooth load. On node add: steal ranges. On failure: replicate to successor
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 2: Durability
_Sync replicate to 3 AZs before 200 OK on PUT. Background scrub detects bit rot_
> [!CAUTION]
> **🔴 Weak** — Oversimplify durability — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Sync replicate to 3 AZs before 200 OK on PUT. Background scrub detects bit rot
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 3: Large objects
_Multipart with part ETags. Coordinator commits manifest atomically_
> [!CAUTION]
> **🔴 Weak** — Oversimplify large objects — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Multipart with part ETags. Coordinator commits manifest atomically
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 4: Listing at scale
_Prefix index per bucket shard. Paginate with continuation tokens_
> [!CAUTION]
> **🔴 Weak** — Oversimplify listing at scale — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Prefix index per bucket shard. Paginate with continuation tokens
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Object store script.
2. "Separate metadata path from data path — never stream gigabytes through metadata DB."
3. "Consistent hash ring places objects. Three replicas across failure domains."
4. "Multipart for large uploads. Presigned URLs for direct client ↔ data node transfer."
Whiteboard
```
Client -> API -> Metadata DB (bucket/key -> node list)
-> Data nodes (replicated chunks)
```
Metadata is the control plane; data nodes are the data plane.
Digital wallet (Apple Pay) Hard
Tokenization · PCI scope · Double-entry ledger
HSMToken vaultLedger3DSIdempotencyPCI DSS
Card numbers are tokenized in a HSM-backed vault — merchants never see PAN. Payments authorize against the payment network with device cryptograms. A double-entry ledger records all money movement immutably.
Problem
Design a digital wallet: add cards, pay in stores/apps, P2P transfers, with PCI compliance and financial correctness.
Hard parts: tokenization, fraud, and ledger integrity.
Failures
**Double tap charge**
Duplicate authorization.
_Fix:_ Idempotency key per tap. Network-level dedup token.
**HSM unavailable**
Cannot decrypt tokens — payments fail.
_Fix:_ HSM cluster with failover. No software fallback for PAN ops.
**Ledger imbalance**
Money created/destroyed.
_Fix:_ Batch invariant job. Freeze accounts on mismatch.
Estimation
| Field | Value |
|-------|-------|
| Assumptions | 500M wallets, 5 tx/user/day, $25 avg |
| Read QPS | Balance reads: 500M×2/86400 ≈ 12K/s |
| Write QPS | 5×500M/86400 ≈ 29K auth/s peak |
| Storage | Event log grows ~100B events/year — cold archive |
| Cache math | Active balance cache in Redis per wallet |
| Verdict | Ledger sharding by wallet_id. HSM is throughput ceiling. |
Design decisions
**Token per device vs per user**
→ Per device token
Stolen phone does not expose other devices.
_Revisit when:_ Per user if simplifying.
**Ledger sharding**
→ Shard by wallet_id
P2P between wallets may need saga if different shards.
_Revisit when:_ Single shard for interview MVP.
**Offline payments**
→ Secure element counter
Limited offline quota. Sync when online.
_Revisit when:_ Online-only if out of scope.
Follow-up Q&A
**How is PCI scope reduced?**
Merchant never sees PAN — only tokens. Vault is isolated. SAQ A scope for merchants.
**How do P2P transfers work?**
Debit sender ledger, credit receiver in one transaction if same shard; else saga with hold.
**How do refunds work?**
New ledger events reversing original. Link refund_id to payment_id.
**How do you handle currency conversion?**
FX rate at auth time stored on event. Settlement may differ — reconcile FX gain/loss.
**How do you detect fraud?**
Velocity, device attestation, geolocation mismatch, ML risk score.
**How do you support recurring billing?**
Merchant-specific token + mandate record. Network initiates charge with idempotency.
**How do you audit the ledger?**
Immutable log + daily sum(debits)+sum(credits)=0 check.
**How does this differ from Stripe?**
Stripe is merchant acquirer platform. Wallet is consumer token vault + pass-through auth.
Evolution
**v1 — Token + auth** — HSM vault. Network auth. Simple ledger.
**v2 — P2P + ledger** — Double-entry. Idempotency. Fraud rules.
**v3 — Global wallet** — Multi-currency, offline tap, network token lifecycle, regulatory reporting.
Why it's hard to scale
Financial correctness and PCI boundaries matter more than raw QPS.
Key points
- **Tokenization** — Replace PAN with device-specific token. HSM generates and stores mapping.
- **PCI scope reduction** — Merchant handles tokens only. Vault is isolated PCI zone.
- **Double-entry ledger** — Every transfer = balanced debit/credit events. Immutable log.
- **Idempotent payments** — client_request_id dedup prevents double tap charges.
- **3DS / biometrics** — Step-up auth for high-risk transactions.
- **P2P transfers** — Internal ledger move before external ACH settlement.
> Token vault + immutable ledger + idempotent authorization.
Tradeoffs
**Store PAN vs token only** — Token-only is mandatory for PCI. PAN in HSM only.
**Sync vs async settlement** — User sees auth result sync. Settlement with network async.
**Ledger event sourcing vs balance table** — Event log is audit-proof. Balance is derived/cache.
> "Tokens not PANs, HSM vault, double-entry ledger, idempotent auth requests."
Deep dives
#### Deep dive 1: Tokenization and HSM
> [!CAUTION]
> **🔴 Weak** — encrypt PAN in DB
>
> [!WARNING]
> **🟡 Strong** — HSM generates tokens; PAN never leaves secure enclave
>
> [!TIP]
> **🟢 Staff+** — Name the metric you'd alert on and when you'd revisit this design.
#### Deep dive 2: Ledger correctness
_Append-only events. Daily balance invariant check. No in-place balance updates_
> [!CAUTION]
> **🔴 Weak** — Oversimplify ledger correctness — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Append-only events. Daily balance invariant check. No in-place balance updates
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 3: Fraud
_Velocity limits, device fingerprint, ML risk score. Step-up 3DS above threshold_
> [!CAUTION]
> **🔴 Weak** — Oversimplify fraud — name one component, skip failure modes and metrics.
>
> [!WARNING]
> **🟡 Strong** — Velocity limits, device fingerprint, ML risk score. Step-up 3DS above threshold
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
#### Deep dive 4: Offline tap
_Stored cryptogram on secure element. Limited offline spend counter_
> [!CAUTION]
> **🔴 Weak** — One global INCR key for all traffic.
>
> [!WARNING]
> **🟡 Strong** — Stored cryptogram on secure element. Limited offline spend counter
>
> [!TIP]
> **🟢 Staff+** — Name metric + revisit trigger when they push depth.
Interview script
1. Wallet script.
2. "Card enrollment tokenizes PAN in HSM — app never stores PAN."
3. "Payment: token + cryptogram to network. Idempotency key on every tap."
4. "Ledger records auth/capture/settle as immutable events."
Whiteboard
```
App -> Wallet API -> Token Vault (HSM)
-> Auth svc -> Payment network
-> Ledger (event log)
```
Vault and ledger are isolated trust boundaries.