9.1The tables▶ 83:04

An important point: you book a room type (for example Double queen), not a specific room; the room number is assigned at check-in.

Why are transactions in their own table? Because a booking goes through stages: it is booked (the first transaction), then the user may change the dates so there is a price difference to pay or refund (a second transaction), or cancel so there is a refund with a negative amount (a third transaction).

9.2Room type inventory▶ 86:34

Inventory is tracked per hotel, per room type and per day, not per physical room. This gives us:

• Bulk availability updates.
• Pricing and search that scale.
• Simple overbooking logic.

There is a row for each (hotel, room type, date). We pre-populate it two years ahead, and a daily cron job adds new dates as time passes.

(Explainer's note): The speaker showed sample data for December 15, 16 and 17 without reading out every number. The IDs and the total_reserved figures in this table are a (suggested example), except that 100, 92 and 87 were mentioned in the explanation (100 and 92 in the December 23–27 availability example, 87 in the cache section), not tied to these dates, so assigning them to December 16 and 17 is also a suggested example.

9.3Availability checks and overbooking▶ 90:33

The speaker's example: a booking from December 23 to 27:

SELECT date, total_inventory, total_reserved
FROM room_type_inventory
WHERE hotel_id = :hotel_id
  AND room_type_id = :room_type_id
  AND date BETWEEN '2025-12-23' AND '2025-12-26';

And the condition for each day:

total_reserved + rooms_to_book <= total_inventory

Example: total_inventory = 100 and total_reserved = 92, so 8 are left. The user wants one room: 92 + 1 = 93 ≤ 100, so the booking is allowed.

With overbooking: if we allow 10%, we multiply the inventory by 1.1, so with 100 rooms we allow bookings up to 110, because we expect about 10 to be cancelled ahead of time:

total_reserved + rooms_to_book <= 1.1 * total_inventory

(Explainer's note): The speaker said "Select... where the date is between the range"; writing the SQL in this form, and the year in the date, are mine. I made the last day the 26th because the night of the 27th is the check-out day, which is my assumption, not the video's.

9.4Why a relational database here?▶ 92:09

1. ACID, and especially consistency, to prevent double booking.
2. The system is read-heavy (1,000 QPS versus 17 TPS), relational DBs handle reads well, and we can add Elasticsearch for search.
3. Clear relationships: rooms, hotels, types, reservations and transactions.
4. Complex queries: joins between inventory and rates, filtering by dates, and aggregating totals.

9.5Concurrency and race conditions▶ 93:26

The first scenario (the same user twice): solved by the reservation key. The second attempt gets a duplicate key error, so we return the existing booking.

The second scenario: two different users, and the room type has one spot left.

The default isolation level is read committed: you see data only after other transactions commit. But there is a gap between when you read (check availability) and when you write (book):

sequenceDiagram
    participant A as User A tx
    participant DB as Inventory row
    participant B as User B tx
    A->>DB: read total_reserved = 99 of 100
    B->>DB: read total_reserved = 99 of 100
    A->>A: 99 + 1 fits in 100, OK
    B->>B: 99 + 1 fits in 100, OK
    A->>DB: book, reserved = 100
    B->>DB: book, reserved = 100 (2 bookings)
    Note over DB: double booking

Both read correct data and passed the check, and then each one wrote over the other. That is why read committed alone is not safe for bookings.

(Explainer's note): The speaker says the two transactions "override each other". If each writes the value it computed, both write 100 (a lost update) and two bookings exist for one room, which is what the diagram shows.

9.6The two solutions▶ 95:01

(Explainer's note): "Transactions block each other" is the speaker's description. In PostgreSQL, Serializable (SSI) does not queue transactions; it aborts conflicting ones with a serialization error, and the application must retry them. Either way the cost grows with TPS.

The speaker considers the second solution the better one: FOR UPDATE prevents the race condition by serializing access to those specific rows, and read committed stays PostgreSQL's default. Serializable is acceptable if the TPS is low.

BEGIN;
SELECT total_inventory, total_reserved
FROM room_type_inventory
WHERE hotel_id = :h AND room_type_id = :rt
  AND date BETWEEN :start AND :end
FOR UPDATE;
-- check total_reserved + n <= total_inventory, then:
UPDATE room_type_inventory SET total_reserved = total_reserved + :n
WHERE hotel_id = :h AND room_type_id = :rt AND date BETWEEN :start AND :end;
INSERT INTO reservations (...) VALUES (...);
COMMIT;

(Explainer's note): The speaker named SELECT ... FOR UPDATE, but this full code (suggested example) shows where the lock happens.

10.1Sharding by hotel_id▶ 97:18

With 1,000 QPS and 17 TPS, you will eventually need sharding, and it will show in how long queries take. The most important decision is the shard key.

The speaker's proposal: hotel_id. Why?

• Most queries filter by hotel first (is its location suitable for the search?) and then fetch the room types.
• All of a hotel's data stays on the same shard, so there are no cross-shard joins.
• Local transactions within the shard.

To start: modulo sharding:

shard = hash(hotel_id) % number_of_shards

This is fine when you know the number of shards won't change much. Example: 1,000 QPS over 16 shards is about 62 QPS and about 1 TPS per shard.

10.2Modulo or consistent hashing?▶ 99:07

• Modulo: if you add a 17th shard, you have to redistribute almost all the data in the 16 old ones.
• Consistent hashing: moves only about 1/17 of the data, the part that will be stored on the new shard.

How? A hash ring: servers and keys are placed as points on a circle, and each key goes to the nearest server clockwise. When you add a new server, only the keys between it and the server before it are affected.

The rule: a small, fixed number of shards means modulo. If you will add many shards, use consistent hashing.

flowchart LR
    K1((key A)) -->|clockwise| S1[Server 1]
    K2((key B)) -->|clockwise| S2[Server 2]
    K3((key C)) -->|clockwise| S17[New server 17]

(Explainer's note): The speaker drew the ring on the whiteboard. The diagram here is my simplification of the idea. He also pointed to a video on consistent hashing in his System Design course.

10.3Caching strategy▶ 100:28

We cache the "hot" inventory data in Redis or Memcached:

flowchart TB
    S[User searches] --> C{Redis hit and TTL valid?}
    C -->|yes| Show[Show inventory]
    C -->|miss or expired| DB[(Read DB)] --> Fill[Populate Redis] --> Show
    Show --> Book[User books]
    Book --> V[Validate + write in DB]
    V --> Inv[Invalidate affected cache keys]

The key point: the booking itself is always confirmed against the DB, and once it succeeds we invalidate the affected keys, so the next search sees that this room type is no longer available.

10.4Search with Elasticsearch▶ 103:06

Elasticsearch is a search and analytics engine, a database specialized in search, not for transactional workloads like PostgreSQL and MySQL. SQL is excellent at "get the row where id = 123" or "do an insert", and Elasticsearch is excellent at "find all the documents similar to this text, rank them by relevance, filter by several conditions, and do it fast across millions of documents". Example: hotels in Georgia, priced 80 to 180 a day, rated 8+.

Why not SQL? If you run SELECT * FROM hotels WHERE location LIKE '%Paris%' AND amenities LIKE '%WiFi%' AND description LIKE '%luxury%' ...:

How is the data stored? Yes, we duplicate the data. In SQL the hotel is spread across tables (hotel, room types, amenities), and Elasticsearch turns it into one JSON document with the hotel_id, name and address, with the location and room types nested inside. No joins, which is what makes it fast.

PostgreSQL is the single source of truth. Availability, overbooking and every transaction are confirmed against it. Elasticsearch is for search only.

10.5Syncing with CDC▶ 108:17

To keep Elasticsearch eventually consistent with PostgreSQL:

flowchart LR
    PG[(PostgreSQL<br/>source of truth)] -->|write-ahead log| DBZ[Debezium connector]
    DBZ -->|one topic per table| K[[Kafka]]
    K --> C[ES consumer]
    C -->|update index| ES[(Elasticsearch)]

1. PostgreSQL writes every change to the write-ahead log (WAL).
2. Debezium (an open-source CDC tool) reads that log in real time.
3. It publishes the changes to Kafka, with one topic per table.
4. A consumer reads the events and updates the index in Elasticsearch.

It also helps with crash recovery: even if PostgreSQL goes down, you can read from the transaction log.

(Explainer's note): The auto-generated subtitles say "bases connector" and "dbium"; what is meant is the Debezium connector.

10.6Database replication▶ 109:42

Sharding splits the tables (for example geographically), and replication scales reads: a primary for writes, and read replicas in different availability zones, with each user connected to the nearest replica.

Reading after a recent write: in the query for a given user's reservations, a variable decides whether to use the primary or not:

• If there was a recent write, read from the primary, because the replica may not be updated yet.
• Otherwise, read from the nearest replica.

10.7Global load balancing▶ 111:09

Users come from the US, Europe, Asia and elsewhere, so the sensible algorithm is geo-latency routing by region. There are exceptions (an American travelling in Europe), but in general a European user searches for hotels in Europe.

In each region:

• Read replicas of the DB.
• A region-specific Redis (there is no point caching European hotels in the US region).
• A region-specific Elasticsearch.

But there is one primary (even if it is a shard, for example shard 7), to avoid conflicts from several regions writing to the same data. This works because the TPS is low anyway (17), so one primary per shard and many replicas across regions is enough.

flowchart TB
    subgraph US[US region]
      R1[(Read replica)]
      C1[(Redis)]
      E1[(Elasticsearch)]
    end
    subgraph EU[EU region]
      R2[(Read replica)]
      C2[(Redis)]
      E2[(Elasticsearch)]
    end
    P[(Single primary<br/>per shard)] -.->|replicate| R1 & R2
    W[All writes] --> P

(Explainer's note): In the intro the speaker said he would talk about failures under load and recovery in the hotel system, but this part ends at replication and global routing without a section on failures. Failures are covered in detail in WhatsApp (section 15).

Messaging apps come up a lot in senior and staff interviews because they force you to deal with: stateful connections, real-time delivery, consistency, scale, failures of parts of the system or full disaster recovery, and multi-device sync (mobile and desktop). What is expected is that you show you understand not just the happy path, but also everything that can break while messages fly between users who are not connected over mobile networks.

The plan: a simple working version for a few thousand users, then scale to billions, then find where it breaks and fix it properly.

11.1Requirements▶ 114:45

Functional:

• One-on-one messages.
• Group chats of up to 100 participants.
• Delivery when the user is online and also when they are offline.
• Attachments: images, video and audio (voice notes).
• Message states: sent, delivered and read.
• Online/offline status and last seen.

Non-functional:

• Latency under 500 ms when both are online.
• Guaranteed delivery: messages must not disappear.
• Billions of users and high throughput.
• We don't keep messages on the server longer than necessary, so storage stays reasonable.
• Resilient: if a component goes down (such as the server that is supposed to deliver the message), the user eventually receives it when the server comes back.

11.2Estimating scale (back-of-the-envelope)▶ 116:24

An important point: most messages are delivered and deleted from the server within seconds. Users download messages to their devices and back them up if they want (for example to Google Drive), and the server gets cleaned up. The speaker assumes 30-day retention for undelivered messages, and that about 10% of users keep their history on the servers. This is what lets WhatsApp run on little infrastructure.

(Explainer's note): 115,000 × 5 ≈ 575,000, and the speaker rounded the peak to 350–500 thousand. Both are rough estimates; what matters is the order of magnitude: hundreds of thousands of writes per second.

11.3High-level design▶ 119:03

flowchart LR
    App[Mobile app] --> LB[L4 load balancer]
    LB --> CS[Chat servers<br/>WebSocket]
    CS --> Q[[Kafka / RabbitMQ]]
    Q --> MS[Message storage svc]
    MS --> DB[(Cassandra / DynamoDB)]
    CS --> R[(Redis<br/>who is where)]
    CS --> N[Notification svc<br/>APNs / FCM]
    CS --> P[Presence svc]
    App -.->|media| Blob[(Blob storage)] -.-> CDN[CDN]

(Explainer's note): The diagram's layout is mine; the components are from the speaker.

11.4Polling, long polling or WebSocket?▶ 123:38

WebSocket's trade-offs:

• It needs load balancers that support it: Layer 4 instead of Layer 7.
• More memory per connection on the server, because it stays open.
• But it is worth it for real-time delivery and the latency requirements.

11.5Managing the chat servers: service discovery▶ 127:39

In production, chat servers crash, get deployed, and scale up and down. The load balancer has to know which ones are up (for example 1 and 2 are healthy, and 3 is throwing timeouts or errors).

The solution: a service discovery cluster (a service registry such as Consul or AWS Cloud Map):

1. When a server comes up on a given IP, it registers itself in the registry.
2. The registry runs a health check every 10 seconds on each server.
3. The load balancer asks for the list of healthy servers, and the registry sends it (for example only 1 and 2).
4. The load balancer updates its routing table and sends nothing to server 3.

12.1The tables▶ 129:36

(Explainer's note): The speaker says the registry is refreshed about every 300 seconds "with each heartbeat", but in 12.2 he puts the heartbeat at every 5–10 seconds. The two numbers don't match; treat both as rough examples.

12.2The API: WebSocket and REST▶ 132:19

Over the WebSocket:

Media over REST:

1. POST /api/media with the file type, size and conversation.
2. The server responds with an upload URL (pre-signed), a media_id and an expiry time.
3. The client uploads the file to the pre-signed URL directly on the media storage, which responds with upload success.
4. The message is sent with the media, and the server acks it. If an error happens, it responds with message delivery failed and the reason in the body.

(Explainer's note): The auto-generated subtitles say "the onlet count" in the auth success response, and the word is unclear. The closest reading is "unread count", but that is my guess.

12.3Multi-device sync▶ 135:35

The speaker's example: user A has 3 devices (mobile, tablet and desktop), and user B has two devices (mobile and laptop), and each device may be connected to a different chat server.

1. A's mobile sends to server A, and the server replies with Sent.
2. Server A asks Redis: "where is B?", and it replies that B has two devices, on server B and server E.
3. The same message shows as Sent on A's tablet (through server C) and A's desktop (through server D).
4. Server B delivers it to B's mobile, and server E to B's laptop.
5. B's mobile acks, so server B notifies server A, and A sees Delivered.
6. B reads it, so a read receipt goes to server B, then to server A, and A sees Read; it is also marked Read on B's laptop.

Conflicts: last write wins using version numbers.

Optimizations:

• Incremental sync: a new device doesn't need the whole history. It requests the last 30 days, and older messages are fetched when the user scrolls up.
• Very old messages live in cold storage and are lazy loaded.

The trade-off: about double the delivery work, because each message goes to more than one device. But it is a core feature, because users want the same conversations on mobile and desktop.

12.4How a message travels: online and offline▶ 140:16

Case one: both are online

sequenceDiagram
    participant A as User A
    participant SA as Chat server A
    participant Q as Kafka
    participant R as Redis
    participant SB as Chat server B
    participant B as User B
    A->>SA: send via WebSocket
    SA->>SA: generate timestamp-based message ID
    SA-->>A: ack - sent
    SA->>Q: push for persistence
    SA->>R: where is B?
    R-->>SA: B is on server B
    SA->>SB: route via Redis pub/sub
    SB->>B: deliver in real time
    B-->>SB: delivery ack
    SB->>SA: forward delivery
    SA-->>A: status delivered
    Note over Q: queue writes to Cassandra

When B opens the message, the same steps repeat and the status becomes Read. Optionally, we put the message on the queue for analytics or archiving asynchronously.

Case two: B is offline

1. The same start: A sends, and server A asks Redis.
2. B is offline, so the message goes into B's inbox in Redis, and also into an inbox table in the DB.
3. Server A calls the notification service, which sends a push through APNs or FCM.
4. The message is put on the queue, so it gets stored in the DB.
5. When B comes back online and opens a WebSocket with, say, server C: the server pulls the pending messages from the Redis inbox, and any missed messages from the DB inbox, delivers them, and notifies A of the status.

12.5Group messages (fanout)▶ 145:02

The speaker's example: group 789 with about 50 members.

1. A sends to the group on their chat server.
2. The server asks the group service: who is in group 789? It returns the list of user IDs.
3. It checks the status of the 50 in Redis (so it doesn't waste DB queries): 30 online and 20 offline.
4. For the 30 online: it publishes on Redis Pub/Sub (a channel per user), so delivery happens in parallel. For example 15 on server B and 15 on server C, and each server delivers over the WebSocket. The acks return to server A so it can update the status for A.
5. For the 20 offline: the same idea as one-on-one: it writes to the Redis inbox and 20 inbox records in the DB, and a push notification for each one.
6. The message is stored only once: one write from server A to the queue, and the queue writes it to the DB.
7. A sees: delivered to 30, pending for 20.

12.6Media: pre-signed URLs and a CDN▶ 148:54

The wrong way: the client uploads the image to the chat server, the server stores it, and the recipient downloads it from the chat server. This is not scalable: the servers become bound by the bandwidth of moving files, and they are doing what they are bad at (moving large blobs) instead of what they are good at (routing messages quickly).

The right way:

sequenceDiagram
    participant A as App A
    participant SA as Chat server A
    participant S3 as Blob storage
    participant SB as Chat server B
    participant B as App B
    participant CDN
    A->>SA: request pre-signed upload URL
    SA->>S3: generate upload URL
    SA-->>A: upload URL
    A->>S3: direct upload
    S3-->>A: media URL
    A->>SA: chat message with media URL
    SA->>SB: deliver message
    SB->>B: message + media URL
    B->>SB: tap to view - request download URL
    SB-->>B: pre-signed download URL
    B->>CDN: download via CDN
    CDN-->>B: stream media

Downloading: two ways:

• Directly from storage: simpler.
• Through a CDN: extra steps, but the performance and cost gain is huge at this scale. So the CDN is usually the better solution.

Why does this matter? A single chat server holding 100,000 connections could go down from just 100 users uploading small files of a few megabytes at the same time. This way the chat servers move only small messages containing links, and the blob storage and CDN carry the heavy bandwidth.

TTL: pre-signed URLs expire, and the media itself has a TTL of about 30 days matching the message retention, after which it is deleted automatically.

13.1Kafka partitions and storage instances▶ 154:26

To scale further, we decouple the servers from each other with a message queue such as Kafka:

• The message storage service is stateless, so we run as many instances as we need.
• Each instance takes partitions (for example 0–9, 10–19 and 20–29).
• Each instance has a Kafka consumer that does batch writes to the database cluster.
• On failure: retry after a timeout. If it still fails: we don't delete the message from the queue so it isn't lost, or we put it in a dead letter queue (a separate queue) and process it later.
• After the write: an index updater updates the indexes for queries.

13.2The pipeline in detail, and batching▶ 156:36

flowchart LR
    Q[[Kafka]] -->|batch of 500, poll every 100ms| C[Consumer]
    C --> TP[Thread pool<br/>20 workers]
    TP --> V[Validate schema<br/>+ size checks]
    V --> E[Enrich: dedup hash,<br/>mentions, links, indexes]
    E --> BW[Batch writer<br/>1000 msgs or 5 s]
    BW -->|sort by shard,<br/>group by conversation| DB[(Cassandra)]
    DB --> IU[Index updater] --> OK[Mark processed]
    BW -.->|failed| RT[Retry] -.->|still failing| DLQ[[Dead letter queue]]

1. Batch size of 500 messages, and the consumer polls every 100 ms.
2. Parallel processing with a thread pool of 20 workers.
3. For each message: schema validation, and metadata checks (media size; if over the limit we respond with an error).
4. Enrichment: a hash for deduplication, extracting mentions (@username) and links for the preview, and preparing the indexes.
5. Batch writer: collects 1,000 messages or every 5 seconds in memory. When it is full: it sorts by target shard, groups by conversation, and prepares the batch.
6. An atomic batch insert into the cluster, then the index updater, then marked as processed.
7. If part of it fails (for example 10 out of 1,000): a simple retry first (maybe a rate limit or something minor), then the dead letter queue, with several retry steps (up to about 10) before we notify the user that the message wasn't sent.

Why batching?

He shows an example of a batch write in Cassandra:

BEGIN BATCH
  INSERT INTO messages (timestamp, sender_id, receiver_id, content, media_url, ...) VALUES (...);
  INSERT INTO messages (...) VALUES (...);
  -- ... up to 1,000 messages
APPLY BATCH;

(Explainer's note): This code is my condensed rewrite of what the speaker showed on screen. He described the batch as a "transaction".

(Explainer's note): The 5 ms and 50 ms figures are the speaker's illustration. In Cassandra, large batches that span many partitions are generally an anti-pattern for throughput; batching pays off when the batch is grouped by partition, which is why the writer sorts by shard and groups by conversation.

13.3End-to-end encryption▶ 161:43

In WhatsApp, messages are encrypted on the sender's device and decrypted only on the recipient's device. Even the servers (A and B) can't read the content. The solution: the Signal protocol.

1. Each user generates an identity key pair at registration: the public key is uploaded to a key server, and the private key never leaves the device.
2. B also uploads prekeys, for example 100 signed prekeys, and the server stores them for anyone who will start a conversation.
3. When A messages B for the first time: A fetches from the key server B's identity key and one of the prekeys.
4. A generates a key pair and performs a key exchange, so both end up with the same session key.
5. Each message is encrypted with the session key, along with a message counter that increments with every message. So even if the session key is stolen, decrypting every message is still very hard.

(Explainer's note): The counter claim is the speaker's. A counter alone does not protect messages once a session key is stolen; in the real Signal protocol that protection comes from the Double Ratchet, which derives a new key for every message.

sequenceDiagram
    participant A as User A device
    participant KS as Key server
    participant B as User B device
    A->>KS: upload public identity key
    B->>KS: upload public identity key + 100 signed prekeys
    A->>KS: fetch B identity key + one prekey
    A->>A: key exchange - derive session key
    A->>B: encrypted message + counter, via chat servers
    B->>B: decrypt on device

(Explainer's note): In one sentence the speaker said messages are "encrypted with A's public key and decrypted with A's public key". That is a slip of the tongue; in the Signal protocol both sides arrive at a shared session key from the key exchange and encrypt with it, which is what he said in the very next sentence. And a public key alone can't decrypt something encrypted with it.

13.4Rate limiting and abuse prevention▶ 165:08

Without rate limits, any hacker could do scraping or a DDoS. So each action type has a per-user limit:

The speaker says these numbers are examples, and they are decided during the blueprinting phase.

When the limit is exceeded: we block the action and tell the user ("Too many requests, try again in 60 seconds"), or ask for a CAPTCHA.

Progressive penalty:

1. A warning.
2. A one-hour restriction.
3. A 24-hour restriction (like wrong password attempts).
4. Account suspension until a human reviews it: a bot? A hacker? Or a user who didn't understand? They contact the user and reopen the account with the same limits, perhaps with a tag marking it as suspicious.
5. If it repeats: it may be permanently deleted.

Token bucket (from the code he showed): each action has a capacity and a refill_rate, and on every request:

class TokenBucket:
    def __init__(self, capacity, refill_rate):
        self.capacity, self.refill_rate = capacity, refill_rate
        self.tokens, self.last_refill = capacity, time.time()

    def allow(self):
        self.refill()                     # add tokens based on time since last_refill
        if self.tokens >= 1:
            self.tokens -= 1
            return True                   # continue: store message / create group
        return False                      # block, notify, maybe CAPTCHA

Why a central Redis? Each user is connected to a different server over the WebSocket, so the counters must live in a central store shared by all servers (Redis fits because we keep them for up to 24 hours). Atomic operations guarantee accuracy, and there is no way to do server hopping: connecting to another server in the hope that the limit doesn't apply there.

(Explainer's note): This code is my simplification of the logic of the code shown on screen (capacity, refill rate, tokens and last refill time), not a literal copy of it.

14.1How do you choose the shard key?▶ 172:36

The speaker stresses that what the interview expects is not to just pick one, but to present the options and the trade-offs, and say which one you will go with and why.

14.2The solution: shard by user ID with duplicated writes▶ 174:15

• A's conversations are on shard A. When A talks to B (who is on shard B), we also write a copy to shard B.
• The drawback: double the writes.
• The advantage: very fast reads: each user reads only from their own shard.
• Also, each shard has a replica, and each user reads history from the replica, so the primary is free for writes.

flowchart LR
    A[User A writes] --> SA[(Shard A primary)]
    A -->|duplicate write| SB[(Shard B primary)]
    SA -.->|async| RA[(Replica A)]
    SB -.->|async| RB[(Replica B)]
    RA -->|A reads history| UA[User A]
    RB -->|B reads history| UB[User B]

The rule: sharding (splitting the DB into smaller parts) improves writes, and replication improves reads.

The details: for example, user IDs 1 to 1,000 on shard 1. All writes go to the primary node, and from it async replication to 3 replicas (eventually consistent). Reads (opening messages, unread messages) come only from the replicas, and the primary handles writes only.

Interview tip: this trade-off (double writes for fast reads) has to be settled from the requirements: is the system optimized for writes or for reads? If you didn't ask at the start, ask now instead of guessing, and depending on the answer: replication, message ID sharding, or timestamp sharding.

14.3Cleanup jobs▶ 178:02

The system fills up with garbage that will never be read again: expired media, old undelivered messages, and dead (stale) connections.

flowchart LR
    SCH[Job scheduler<br/>Airflow or cron, daily/weekly] --> JQ[[Job queue<br/>separate from message queue]]
    JQ --> WP[Worker pool<br/>containerized]
    WP -->|ack or requeue on failure| JQ
    WP --> T1[Delete expired media]
    WP --> T2[Remove undelivered leftovers]
    WP --> T3[Close stale connections]
    WP --> MON[Monitoring: job results]
    MON --> SCH

• A job scheduler (such as Airflow or a cron job), daily or weekly, that tracks job status through monitoring.
• A job queue for scheduled tasks, separate from the message queue.
• A worker pool: executors in containers that pull jobs, ack them, and put them back on the queue if they fail.
• Results go to monitoring, because this is not something you do once and forget; it has to improve over time.

The trade-off: scheduled jobs add operational complexity (another big thing to monitor and debug). The alternative is to let the garbage pile up until you are forced to do it anyway. So regular cleanup is essential for the system's long-term health.

The happy path (all services up and everyone happy) never happens in reality. There are failures in every part.

15.1Redis is down: fall back to the database▶ 180:47

Redis holds the inboxes of undelivered messages. What if the whole cluster goes down?

1. Writes: the server writes the undelivered message to the Redis inbox (the fastest), and also to the database as a durable backup.
2. Reads: it tries Redis first. If it is empty or unavailable, it falls back to the DB.
3. Recovery: when Redis comes back, a background job syncs it:

SELECT * FROM messages_inbox WHERE delivered IS NULL;
-- for each: write to Redis inbox, keep the DB record

Then we return to normal operation (writing and reading from Redis).

def get_undelivered_messages(user_id):
    try:
        messages = redis.get_inbox(user_id)           # fastest path
        if len(messages) > 0:
            return [parse(m) for m in messages]
    except RedisError:
        log("redis unavailable")                      # e.g. CloudWatch, for tracing
    return db.query("SELECT ... FROM inbox WHERE user_id = ? AND delivered IS NULL", user_id)

Slower, but better than the whole operation failing.

(Explainer's note): This code is my simplification of the logic the speaker explained (try Redis, log, fall back to a select from the DB), not a literal copy.

15.2Regions and global deployment▶ 184:28

Users are everywhere in the world, so we use regions (such as us-east-1 and us-west-2 on AWS), and every region has a full stack: chat servers, Redis clusters and database shards, so it works on its own.

The speaker's example: user A in Tokyo opens the app:

1. A DNS lookup connects them to the nearest IP, which is the Singapore region.
2. The region's load balancer, then a chat server over the WebSocket.
3. The number of chat servers varies by region (for example Ireland has many users so more servers, and Singapore fewer).
4. The region's Redis knows where B is, so the message is sent to B's region.
5. User B in London, same story: DNS, then the nearest IP, then the load balancer, and they receive the message from the region they are connected to.

15.3Kafka is down▶ 186:34

Kafka itself is very scalable, but you have to plan for it going down. Inside the cluster there are brokers (servers), the topics are split into partitions, and each partition has a leader and replicas. Producers write to the leader, and the replicas are for redundancy.

The publish logic with the buffer:

def publish_message(msg):
    try:
        kafka.publish(msg)                      # happy path
    except KafkaError as e:
        log(e)                                  # find out why later
        if len(local_buffer) >= 100_000:
            local_buffer.pop_oldest()           # drop oldest, never fail the whole operation
        local_buffer.append(msg)
        disk_log.append(msg)                    # survives a chat-server crash
        # retry interval must cover the time to spin up a new Kafka (e.g. ~2,000 ms)

• The buffer is bounded (for example 100,000 messages). If it fills up, we don't fail the operation; we drop the oldest.
• The retry interval must be longer than the time a new Kafka takes to come up. You need to know that number exactly (the speaker's example: a retry interval of about 2 seconds = 2,000 ms).

Recovery after Kafka is wiped completely:

1. The message storage service requests messages after the last ID.
2. It compares them with the messages actually stored in the DB.
3. It republishes the missing ones to the new Kafka.
4. Once everything is written to durable storage, it returns to normal consumption.

The goal: prevent, or at least reduce, data loss when many parts fail together.

(Explainer's note): This code is my simplification of the logic of the code shown on screen, not a literal copy. The numbers (100,000 and 2,000 ms) are the speaker's examples. The 2 seconds is the retry interval in his illustration, not a realistic time to bring up a whole new Kafka cluster, which in practice takes minutes.

15.4Feature flags and gradual rollout▶ 191:34

In large systems you don't ship a new feature to all users at once; you do a gradual rollout with a feature flag store:

• All the cloud providers have solutions (Azure, AWS), and there are external tools such as LaunchDarkly (the speaker says they used it at his previous company).
• They are key-value pairs: for example is_new_theme_enabled = true/false, and the value differs by environment: true in staging and false in production. Developers see the feature, users don't.
• In production: true for only 20% of users (beta), and if there are no failure logs we open it to everyone.
• The chat server queries the flag store for the available features (per user or per group), sends messages with the new features to apps above a certain version (for example 2.2), and strips the unsupported fields for older apps.