Design a Notification System

A notification system sends timely messages to people: breaking news, a payment receipt, a shipping update. It sends them in three forms — mobile push (the little alert that pops up on your phone), SMS (a text message), and email. At first it sounds easy: "just call an API and send a message." But the hard parts are scale and reliability: how do you take one event and send it across all three channels, how do you handle sudden bursts of traffic, and how do you make sure you never lose a single notification even when an outside service breaks?

Step 1: Ask questions and estimate the scale

Notification questions are left vague on purpose. Nail down the scope first:

• Channels: push notification, SMS, and email.
• Devices: iPhone, Android phone, and laptop/desktop.
• Real-time? Soft real-time — send it as fast as you can, but a small delay under heavy load is okay.
• Triggers: notifications come from client or server services, and can also be set up ahead of time on the server.
• Opt-out: yes — if a user turns off a channel, they stop getting messages on it.

Back-of-the-envelope estimation

The interviewer tells us how many messages go out each day per channel:

Push notifications : 10,000,000 / day
SMS messages       :  1,000,000 / day
Email              :  5,000,000 / day
-----------------------------------------
Total              : 16,000,000 notifications / day

Average send rate  = 16M / 86,400 s ≈ 185 notifications/sec

185 per second is not a lot on average. But the traffic comes in bursts — a marketing blast or an incident alert can spike to many times the normal rate in seconds. And SMS and email are slow calls to outside companies that sometimes fail. Those two facts — bursts plus slow outside services — are the real reason we need message queues and background workers. It is not the raw number of requests per second.

Step 2: How each channel actually sends

Every channel works the same way — your provider builds a message and calls an outside service, which delivers it to the device — but the players differ:

Where contact info comes from: when a user installs the app or signs up, the API servers collect their contact info and save it. Email and phone go in the user table; device tokens go in a separate device table (one user can have many devices).

Step 3: High-level design

The naive version (and why it breaks)

Start with one notification server that takes the API call, builds the message, and calls each provider directly. It has three big problems an interviewer wants you to point out:

• Single point of failure — one server means one thing that can crash and take everything down.
• Hard to scale — the database, cache, and the send logic for every channel are all jammed into one box, so you cannot grow them separately.
• Performance bottleneck — building messages and waiting on slow outside services ties up the server. One slow provider drags everything else down with it.

The improved design

Three changes fix all three problems:

1. Move the cache and database out of the notification server.
2. Run many notification servers behind a load balancer that adds more servers automatically as load grows.
3. Add message queues to decouple the pieces — one queue for each notification type.

Service 1..N ─▶ Notification servers ─┬─▶ iOS PN queue   ─▶ iOS workers   ─▶ APNS     ─▶ iOS
   (triggers)   (API, auth, validate, │   Android queue  ─▶ Android workers─▶ FCM      ─▶ Android
                 template, rate-limit) ├─▶ SMS queue      ─▶ SMS workers   ─▶ Twilio   ─▶ Phone
                                       └─▶ Email queue    ─▶ Email workers ─▶ Sendgrid ─▶ Inbox
                                       ▲
                              Cache + DB (user/device/template, notif log)

Reading left to right:

• Notification servers offer internal-only APIs, check that emails and phone numbers are valid, pull the data they need from the cache and database to build the notification, and then drop the event onto the right queue.
• Cache holds user info, device info, and message templates. Database stores user, notification, and settings data.
• Message queues are waiting lines that soak up bursts and keep the pieces from depending on each other. Each notification type gets its own queue, so when one outside service goes down, the other types keep working.
• Workers take events off the queues and call the matching outside service.

A typical send API looks like this:

POST https://api.example.com/v1/sms/send
{
  "to": "+14155550000",
  "from": "+14155551111",
  "template_id": "back_in_stock",
  "params": { "item_name": "Sneaker X", "date": "2026-07-01" }
}

Step 4: Deep dive — reliability

The single most important rule: a notification system must never lose data. A notification can be late or arrive out of order, but it must never be dropped. Two mechanisms get you there.

Save first, then retry

The notification servers write every event to a notification log database as they put it on the queue, so nothing lives only in memory. When a worker's call to an outside service fails, the event goes back on the queue to try again. If it keeps failing past a set number of tries, an alert pages the developers. This is exactly the SMS-fails-then-retries part in the animation.

Exactly-once? No — remove duplicates instead

Because retries and the spread-out nature of the system can create duplicates, you add a dedupe check on the event ID (a check that spots repeats). When an event arrives, you check whether you have seen that ID before — if yes, throw it away; if no, send it and record the ID. This cuts down duplicates without pretending exactly-once is achievable.

Step 5: Deep dive — the supporting parts

A real notification system is more than just the send path:

The settings check is non-negotiable — it enforces the opt-out rule from Step 1, right at send time:

user_id  bigint
channel  varchar   -- push | sms | email
opt_in   boolean   -- must be true to send this channel

Wrap-up

The finished system is scalable, decoupled, and reliable thanks to a handful of deliberate choices:

• Reliability — save to a notification log and retry from the queue, so notifications get delayed, never lost.
• Decoupling — one message queue per channel; a broken provider hurts only its own channel.
• Respect user settings — check opt-in before every send.
• Rate limiting — cap how often you message someone so they do not tune you out.
• Security — appKey / appSecret so only approved clients can send.
• Tracking and monitoring — track events and watch queue depth at every stage for health and insight.

Practice

Answer these to check you understood the notification system.