Web crawler system design

Questions to ask the interviewer

The design in short

A crawler is a loop: take a link, download the page, keep a copy, find new links, and put them back in the list. The loop is easy. Keeping it fast and cheap, page after page, is the interview.

Waiting is the main cost. Downloading one page takes about a second, and almost all of that second is spent waiting on the other site, not working. A machine that fetches one page at a time is idle most of the time. So ask how many fetches can be running at the same time, and what each one holds while it waits. This is the number that sets how many fetchers you need.

Split the work into steps. The scheduler hands out links, fetchers download, and something else stores the pages. Steps that run at different speeds should not wait on each other. Think about what sits between them, so a slow step does not stop a fast one.

What to draw:

• the list of links still to fetch, and who hands them out
• the fetchers, and how many pages each one can work on in parallel
• where the page copies go: ask which kind of store fits many small files
• how a fetcher learns which link to take next

Do the storage math early. Multiply pages a second by the page size, then by the seconds in a month. The total tells you which kind of store fits, and what it will cost to keep.

Then check the fetchers against the page rate in the brief, with room to spare. Build the loop in the game and you can see where the pages pile up.