Monitoring system design

Questions to ask the interviewer

The design in short

A monitoring service is a write machine first. Every server reports a set of numbers on a timer, day and night, whether anyone looks or not. Graphs and alerts come second, but they are the reason the service exists. The interview is about taking in a steady flood of small numbers and still answering a graph in a moment.

Each number has a time. A point is a name, a few labels, a value and the second it was taken. Points arrive in time order and are almost never changed after. Ask what kind of store fits that shape. A general database keeps rows it can update and join; a store built for points appends them to the end of a line and reads a stretch of that line back.

Count before you choose. The brief gives you the servers, the numbers each one sends and how often. Multiply them out: 1,000 servers with 100 numbers every 10 s is 10,000 points a second. Then count the graph side. A graph with 5 lines over the last hour reads every point in that hour for each line. Compare both totals with what one box can take.

Items to draw on the whiteboard:

• the servers that receive the batches of points and answer graph requests
• the store that keeps the points, and how a graph reads a range of them
• the check that decides a server is in trouble
• the path from that check to the on-call engineer's phone

Alerts are a promise. A graph that loads slowly annoys people. A check that fails and tells nobody is a real outage that stays hidden. The brief asks for the on-call to hear within a minute. Think about where the alert starts, which parts it passes, and what happens to it if one of them is busy.

Stay inside the brief. Logs and traces are related tools with their own trade-offs, and a good follow-up question. Keep this interview to numbers, graphs and the alert path.

Wire the servers to your design in GonnaCrash and watch whether the points, the graphs and the alerts all get through.