Chat message store system design

Questions to ask the interviewer

The design in short

A chat app never forgets. Every message a person sends stays forever. People open a channel, read its newest page, and scroll up for older messages. Nothing is deleted on a schedule, so the history only grows. The interview is about where all those messages live, and how one page comes back fast.

Count it. The brief gives 1,000 messages sent a second and 4,000 history reads a second: four reads for each send. The history is 9 TB today. Answers must come back in under 500 ms on average. Put the three numbers next to each other before you pick a store, and ask which one grows.

What a read needs. A read wants one channel, newest first, a page of 50 messages at a time. Ask which store keeps the messages of one channel together and in order, so a page takes few reads, most often one. Then ask which store makes you fetch the 50 messages one by one, and what that costs at 4,000 reads a second.

Items to draw on the whiteboard:

• the app servers that take sends and reads
• the store, and the key that decides where a channel's messages sit
• how a read finds the newest page of one channel
• where the copies of each message live, and how many a read asks

Groups are not all alike. A chat of three friends and a public group of many thousands share one design. A quiet channel gets a few messages a day; a busy one gets thousands. Ask what the biggest group does to the place that holds its messages.

Stay inside the brief. Search, attachments, edits and deletes are real follow-ups, and each makes a fair question. Keep this interview to sends and history reads.

Wire it in GonnaCrash and watch how many reads each page costs.

The post behind this problem

This problem follows Discord's engineering post, How Discord Stores Trillions of Messages. It shows the answer to every round: play first, then read it. Real systems: Cassandra, ScyllaDB.