Ever wondered how you get that instant ping when someone likes your post or sends you a message? That simple notification you see on your phone is the result of one of the most complex distributed systems in modern software engineering. What appears as a millisecond-fast alert actually involves multiple servers, databases, message queues, and third-party services working in perfect harmony.
After implementing notification systems for production applications serving millions of users, I’ve learned what actually works at scale. Most developers think notifications are simple — send a message, deliver it, done. But when you’re handling millions of notifications per day across email, SMS, and push channels, the complexity explodes.
In this comprehensive guide, you’ll learn:
- How notification systems are architected for scale and reliability
- The exact components that power systems handling billions of messages
- Real production challenges and how to solve them
- Platform-specific quirks of FCM, APNs, and other delivery services
- Code examples and architecture diagrams for building your own system
By the end, you’ll have working knowledge of notification infrastructure that you can apply to system design interviews or real production systems. Let’s dive in.
Table of Contents
- Understanding Notification Systems: The Basics
- The Complete Architecture: How It All Fits Together
- Core Components Explained
- The Message Journey: From Event to Notification
- Handling Multiple Notification Channels
- Scaling to Millions: Production Challenges
- Platform-Specific Implementation Details
- Building Your Notification System: Code Examples
- Common Pitfalls and How to Avoid Them
- Production Best Practices
Understanding Notification Systems: The Basics
A notification system is a distributed infrastructure responsible for delivering timely information to users through multiple channels. At its core, it solves one fundamental problem: how do you reliably send the right message to the right person at the right time through their preferred channel?
What Is a Notification System?
Think of a notification system as a sophisticated messaging pipeline. When you like someone’s post on LinkedIn, that action triggers an event. This event flows through multiple layers of processing before arriving as a notification on their device. The system handles validation, user preferences, channel selection, message formatting, delivery, retry logic, and tracking — all in the span of milliseconds.
Modern notification systems support multiple channels:
Push notifications deliver mobile and desktop alerts via Firebase Cloud Messaging for Android or Apple Push Notification Service for iOS. These maintain persistent connections to billions of devices, which is why individual apps don’t manage these connections directly — your battery would die.
Email notifications handle transactional messages like password resets and order confirmations through services like SendGrid or Amazon SES. These need careful sender reputation management to avoid spam filters.
SMS notifications send time-sensitive alerts through telecom gateways like Twilio. These are expensive per message and face strict regulations, so they’re reserved for critical communications like OTPs and security alerts.
In-app notifications appear inside the application using real-time connections like WebSockets. These are the cheapest to deliver but require the user to have the app open.
Why Notification Systems Are Complex
The complexity emerges from scale and reliability requirements. A system handling 100 notifications per day has completely different challenges than one processing 10 million. Consider these real-world scenarios:
During a flash sale, millions of users need simultaneous notifications. Without proper architecture, your message queues overflow, servers crash, and critical notifications get lost. Production systems use dedicated queues with priority levels to ensure transactional messages always get through, even when marketing campaigns are flooding the system.
User preferences add another layer of complexity. Some users want push notifications but not emails. Others enable quiet hours from 10 PM to 8 AM. Your system needs to respect these preferences while batching similar notifications to prevent overwhelming users. When 50 people like your post, you don’t want 50 separate pings — you want one notification saying “John and 49 others liked your post.”
Channel-specific quirks create maintenance nightmares. Apple restricts silent notifications to once every 20 minutes. Send them more frequently and they get throttled or blocked entirely. Devices in low-power mode ignore normal-priority notifications. Email providers have complex spam detection algorithms that can blacklist your domain if you’re not careful. SMS messages face different regulations in different countries.
The Complete Architecture: How It All Fits Together
A production notification system consists of several interconnected components. Each handles a specific responsibility, and together they create a resilient pipeline that can scale to billions of messages. Here’s how they work together:
High-Level Architecture Overview
The architecture follows a pipeline pattern with clear separation of concerns. Events flow through the system in stages, with each stage handling specific responsibilities and having independent scaling characteristics.
At the entry point, your application services generate notification events. When a user comments on a post, processes a payment, or completes a task, that action creates an event. These events get published to an API gateway or notification service that acts as the system’s front door.
The notification service validates incoming requests, applies rate limiting, and performs initial filtering. It checks if the notification type is enabled, validates the payload structure, and ensures the request has proper authentication. This service operates synchronously with the calling application but immediately hands off to asynchronous processing.
Messages enter a distributed queue system like Kafka, RabbitMQ, or AWS SQS. This decouples notification creation from delivery, allowing your application to return immediately without waiting for actual delivery. The queue provides durability guarantees and enables horizontal scaling of processing workers.
Notification processors pull messages from queues and apply business logic. They fetch user preferences from a database, determine which channels to use, format messages according to templates, and apply notification batching rules. This is where intelligence lives — determining that 50 likes should become one batched notification instead of 50 individual ones.
Channel-specific workers handle actual delivery. Separate worker pools manage push notifications, emails, and SMS because each channel has different performance characteristics and failure modes. Push workers might process 10,000 messages per second while SMS workers handle only hundreds due to carrier rate limits.
External delivery providers complete the final delivery. FCM and APNs maintain persistent connections to billions of devices. SMTP servers route emails through the internet. SMS gateways interface with telecom carriers. Your system sends properly formatted requests to these services and handles their responses.
A tracking and analytics layer monitors the entire pipeline. It logs delivery attempts, captures failures, records user interactions like opens and clicks, and generates metrics for monitoring dashboards. This data feeds back into the system to improve delivery rates and optimize user engagement.
Core Components Explained
Let’s break down each component in detail, understanding not just what it does but why it’s designed that way.
The Notification Gateway
The gateway serves as the system’s API layer. It accepts notification requests from your application services and provides two operational modes: single notification submission and batch submission. For instance, when processing a bulk email campaign, batch mode allows you to submit thousands of notifications in one API call instead of making individual requests.
This component handles authentication, authorization, and initial validation. It enforces API rate limits to prevent abuse and protects downstream systems from being overwhelmed. The gateway maintains service-level agreements by implementing circuit breakers that prevent cascading failures when downstream services are unhealthy.
Production tip: Implement idempotency keys at the gateway level. When a client retries a failed request, the idempotency key prevents duplicate notifications from being created. This is critical for transactional notifications like payment confirmations where duplicates cause user confusion and support tickets.
Message Queue System
The queue provides the backbone of reliability and scalability. Popular choices include Apache Kafka for high-throughput scenarios, RabbitMQ for flexible routing, and AWS SQS for managed simplicity. Each has different trade-offs in terms of message ordering guarantees, throughput characteristics, and operational complexity.
Queues enable asynchronous processing that decouples notification creation from delivery. Your application can create a notification and return to the user immediately. The notification gets delivered in the background, potentially seconds or minutes later depending on queue depth and processing capacity. This prevents slow external services from blocking your application’s critical path.
Message durability ensures notifications survive system crashes. When a message enters the queue, it’s persisted to disk before acknowledgment is sent. Even if all processors crash, messages remain in the queue for processing when systems recover. This provides at-least-once delivery semantics — every notification will eventually be attempted.
Topic-based routing separates different notification channels. Email notifications go to an email topic, push notifications to a push topic, and SMS to an SMS topic. This allows independent scaling and failure isolation. If email delivery is slow, it doesn’t impact push notification delivery because they use separate queues and worker pools.
Notification Processor
The processor contains the business logic that transforms raw events into deliverable notifications. It reads messages from the queue, fetches user preferences from a database, applies notification rules, formats messages using templates, and publishes formatted notifications to channel-specific queues.
User preference handling requires careful design. The processor queries a preferences database to determine each user’s channel settings, quiet hours configuration, and notification frequency limits. This data is cached aggressively because it’s read for every notification but changes infrequently. Redis or Memcached typically serve as the caching layer with cache-aside pattern implementation.
Template rendering converts structured data into formatted messages. A notification template might specify that a like notification should read {{actor_name}} liked your post with different formatting for push, email, and in-app channels. The processor substitutes template variables with actual data and generates channel-specific payloads.
Batching logic prevents notification fatigue. When multiple similar events occur within a time window, the processor combines them into a single notification. This requires maintaining state about recent notifications per user, typically using a time-windowed cache. The batching window and rules vary by notification type — likes might batch aggressively while security alerts never batch.
Channel Delivery Services
Each notification channel requires specialized handling due to platform-specific requirements and constraints. Channel services act as adapters that translate generic notification payloads into channel-specific formats and manage interactions with external delivery providers.
Push notification services manage device token registration and delivery through FCM or APNs. Device tokens must be stored and kept up to date because they can expire or change when users reinstall apps. The service handles token invalidation by processing delivery feedback from FCM and APNs, removing invalid tokens from the database.
Email services integrate with SMTP providers like SendGrid, Mailgun, or Amazon SES. They handle sender domain verification, maintain IP reputation, manage bounce and complaint handling, and process webhooks for delivery events. Email has the most complex deliverability considerations because recipient mail servers make sophisticated spam detection decisions.
SMS services interface with telecom gateways while handling international routing complexities. Different countries have different carriers, regulations, and pricing. The service must select appropriate sender IDs, handle character encoding properly, and manage costs by implementing SMS fallback strategies for failed deliveries.
The Message Journey: From Event to Notification
Understanding the complete notification lifecycle helps you build more reliable systems. Let’s trace a notification from creation through delivery, examining what happens at each stage.
Step 1: Event Generation
When Sarah likes Bob’s post on a social media platform, the application service handling that action creates a notification event. This event contains structured data: the event type (post_liked), the actor (Sarah), the target user (Bob), the post ID, and a timestamp. The service publishes this event to the notification gateway via an HTTP API call.
The API call includes an idempotency key to prevent duplicate notifications if the request is retried. The gateway validates the request payload against a schema, authenticates the calling service, and checks rate limits. If everything passes validation, the gateway returns a 202 Accepted response immediately — it doesn’t wait for delivery.
Step 2: Queue Persistence
The gateway publishes the validated event to a Kafka topic for notification processing. Kafka persists the message to disk across multiple replicas before acknowledging receipt. This ensures the notification won’t be lost even if the entire system crashes at this moment. The message is now durable and will be processed eventually.
Message ordering is preserved within a partition, but notifications for different users can be processed in parallel. Kafka partitions messages by user ID, ensuring all notifications for Bob get processed in order while notifications for other users process concurrently on other partitions.
Step 3: Preference Checking and Processing
A notification processor consumes the message from Kafka. It fetches Bob’s notification preferences from a PostgreSQL database, checking the cache first. Bob has enabled push notifications and in-app alerts but disabled email notifications for likes. The processor respects these preferences and prepares messages only for enabled channels.
The processor checks for recent similar notifications using Redis. It finds that Bob received three other like notifications in the past hour. Instead of creating a fourth separate notification, it updates the existing batched notification to say “Sarah and 3 others liked your post.” This prevents notification fatigue while ensuring Bob stays informed.
Template rendering happens next. The processor loads the like notification template and substitutes variables: actor names, post content preview, and timestamp. It generates separate payloads for push and in-app channels because they have different format requirements and character limits. The push notification has a 178-character limit while in-app can be longer.
Step 4: Channel Routing
The processor publishes formatted notifications to channel-specific queues. The push notification goes to the push_notifications topic while the in-app notification goes to the in_app_notifications topic. Each message includes the formatted payload, recipient device tokens for push, and metadata like priority level and time-to-live settings.
Priority levels affect processing order. Transactional notifications like password resets have high priority and jump to the front of the queue. Marketing notifications have low priority and process during off-peak hours. This ensures critical notifications always get through even when the system is under heavy load.
Step 5: External Delivery
A push notification worker consumes the message and calls the Firebase Cloud Messaging API. It includes Bob’s device token, the notification payload, and delivery options like priority and time-to-live. FCM accepts the request and returns a success response. The worker logs this delivery attempt in a MongoDB database for analytics and debugging.
FCM maintains a persistent connection to Bob’s device. When his phone is online and reachable, FCM delivers the notification within milliseconds. The notification appears on his lock screen and in the notification tray. If Bob’s phone were offline, FCM would queue the notification and deliver it when the device reconnects, respecting the time-to-live setting.
Meanwhile, the in-app notification worker stores the notification in a PostgreSQL table. When Bob next opens the app, the application queries this table and displays new notifications in the UI. In-app notifications don’t require external delivery services because they’re pulled by the application rather than pushed.
Step 6: Delivery Confirmation and Analytics
The system tracks delivery status throughout the pipeline. When FCM confirms delivery, the worker updates the notification record’s status to delivered. If Bob taps the notification, the mobile app sends an analytics event recording the interaction. These events feed into dashboards showing delivery rates, open rates, and engagement metrics.
Failure handling is critical for reliability. If FCM returns an error indicating Bob’s device token is invalid, the worker marks the token as inactive in the database. This prevents future wasted delivery attempts. If FCM is temporarily unavailable, the worker returns the message to the queue for retry with exponential backoff.
Handling Multiple Notification Channels
Each notification channel has unique characteristics, constraints, and best practices. Understanding these differences is essential for building a robust multi-channel notification system.
Push Notifications: Mobile and Web
Push notifications require managing device tokens that identify specific app installations. When a user installs your app, it registers with FCM or APNs and receives a unique token. Your backend must store this token along with the user ID and keep it updated. Tokens can expire or change when users reinstall the app, requiring your system to process delivery feedback and clean up invalid tokens.
Platform-specific payload formats add complexity. iOS push notifications use a different JSON structure than Android. iOS requires specific keys like aps, alert, and badge while Android uses data and notification objects. Your system needs separate formatting logic for each platform, or you can use a normalized internal format and transform it during delivery.
Priority and time-to-live settings affect delivery behavior. High-priority notifications wake sleeping devices and deliver immediately. Normal-priority notifications deliver opportunistically when the device is awake to save battery. Time-to-live determines how long FCM or APNs will attempt delivery if the device is offline. For time-sensitive notifications like OTPs, use short TTL values so they expire quickly.
Silent notifications enable background data sync without disturbing the user. However, Apple restricts these to once every 20 minutes per app. Send them more frequently and iOS throttles or blocks them entirely. This means you can’t rely on silent notifications for time-critical background updates.
Email Notifications: Deliverability and Reputation
Email deliverability depends heavily on sender reputation. ISPs like Gmail and Outlook track your sending domain’s behavior and assign a reputation score. Send too many messages that users mark as spam, and your score drops. Eventually, ISPs start filtering your emails to spam folders or blocking them entirely.
Authentication protocols like SPF, DKIM, and DMARC prove that you’re authorized to send from your domain. Without proper configuration, ISPs treat your emails suspiciously. These DNS-based mechanisms allow receiving mail servers to verify that the sending server is legitimate and hasn’t been spoofed.
Bounce and complaint handling maintains reputation. Hard bounces indicate invalid email addresses that should be removed from your database immediately. Soft bounces suggest temporary issues like full mailboxes — retry these a few times before giving up. Complaints occur when users mark your email as spam. High complaint rates damage reputation quickly, so monitor this metric closely and make unsubscribe links prominent.
Content matters for spam detection. Avoid spam trigger words, maintain a good text-to-image ratio, include a physical address, and provide clear unsubscribe mechanisms. Transactional emails like password resets have different requirements than marketing emails — transactional emails should never include promotional content or you risk deliverability issues.
SMS Notifications: Cost and Compliance
SMS notifications are expensive compared to other channels. Costs vary by country and carrier, ranging from fractions of a cent to several cents per message. This makes SMS suitable only for high-value notifications where the cost is justified — authentication codes, transaction alerts, and critical system notifications.
Regulatory compliance varies by region. The US requires opt-in consent and clear identification of the sender. The EU has stricter GDPR requirements. Some countries require sender ID registration with telecom authorities. Your system needs to track opt-in status per user and enforce compliance rules before sending.
Message encoding affects cost. Standard SMS supports 160 characters using GSM-7 encoding. Unicode messages for non-Latin characters reduce capacity to 70 characters per message. Longer messages get split into segments, with each segment billed separately. A 200-character Unicode message consumes three SMS credits instead of one.
Delivery receipts provide confirmation that the message reached the recipient’s phone. However, delivery receipts don’t guarantee the user read the message — just that the carrier delivered it successfully. Some carriers don’t support delivery receipts at all, so you can’t rely on them for all messages.
Scaling to Millions: Production Challenges
Building a notification system that works for 100 users is straightforward. Scaling that same system to handle millions of users exposes challenges that only emerge at scale. Here are the real production issues you’ll face and how to solve them.
Handling Traffic Spikes
Flash sales, breaking news, and viral events create notification spikes that can be 100x normal load. During a Black Friday sale announcement, every user might receive a notification simultaneously. Without proper architecture, your servers crash and critical transactional notifications get lost.
Message queues provide the primary defense against spikes. When a million notifications arrive in one minute, the queue buffers them and allows workers to process at their sustainable rate. The queue depth grows temporarily but gradually drains as workers churn through messages. Monitor queue depth as a key metric — rapid growth indicates insufficient worker capacity.
Auto-scaling workers respond to queue depth. Configure your container orchestration system (Kubernetes, ECS) to spin up additional worker instances when queue depth exceeds a threshold. Workers scale horizontally because they’re stateless — each worker processes messages independently without coordination. During spikes, you might scale from 10 workers to 100, then back down during quiet periods.
Priority queues ensure critical notifications get through even during spikes. Create separate queues for high-priority transactional notifications and low-priority marketing messages. High-priority queues get dedicated worker pools that aren’t affected by marketing campaign traffic. This guarantees password reset emails arrive quickly even when you’re sending a million promotional notifications.
Managing External Service Rate Limits
External delivery services impose rate limits to protect their infrastructure. SendGrid might limit you to 1,000 emails per second. Twilio might cap SMS at 50 messages per second. APNs allows about 2,000 connections per second. Exceed these limits and you get throttled or temporarily banned.
Token bucket rate limiting enforces constraints at your side. Before calling an external API, check if you have available tokens in your bucket. If not, wait until the bucket refills. This prevents your workers from overwhelming external services and getting blocked. Implement separate buckets for each external service since they have different limits.
Worker pool sizing must respect rate limits. If SendGrid allows 1,000 emails per second and each worker sends 10 emails per second, you can run at most 100 workers. Running more workers just causes them to sit idle waiting for rate limit tokens. Right-size worker pools based on external service capacity, not your internal processing capability.
Batch API calls when possible. Instead of calling FCM once per notification, group 100–500 notifications into a single batch request. This reduces API overhead and helps you stay within rate limits. However, batching adds latency — notifications wait until the batch fills or a timeout expires. Balance batch size against acceptable delivery delay.
Ensuring Reliability with Retry Logic
External services fail regularly. Network blips cause timeouts. Services have outages. Rate limits get exceeded despite your best efforts. Your system needs sophisticated retry logic to handle these transient failures without creating duplicate notifications or losing messages.
Exponential backoff prevents retry storms. After a failure, wait 1 second before retrying. If that fails, wait 2 seconds, then 4, 8, 16, up to a maximum. This gives failing services time to recover while ensuring you eventually retry. Add jitter by randomizing wait times slightly to prevent synchronized retries from multiple workers.
Maximum retry limits prevent infinite retry loops. After 5 or 10 failed attempts, move the message to a dead letter queue for manual investigation. Some failures are permanent — invalid device tokens, disabled email addresses, non-existent phone numbers. Retrying these forever wastes resources without achieving delivery.
Idempotency prevents duplicate deliveries. When a request times out, you don’t know if it succeeded or failed. The external service might have processed it even though you didn’t receive a response. Include a unique idempotency key in each request so the service can detect and ignore duplicate submissions.
Dead letter queues capture permanently failed messages. These contain valuable debugging information — why did this notification fail repeatedly? Dead letter analysis often reveals configuration issues, bugs in template rendering, or systematic problems with specific notification types. Monitor dead letter queue depth as a key reliability metric.
Database Performance at Scale
Every notification requires database access to fetch user preferences, device tokens, and notification history. At millions of notifications per day, database queries become a bottleneck unless you architect carefully.
Read replicas distribute query load. User preferences and device tokens are read-heavy but written infrequently. Route these reads to dedicated replica databases that lag slightly behind the primary. This prevents read traffic from impacting write performance on the primary database.
Caching layers reduce database hits dramatically. User preferences change rarely — cache them in Redis with a 1-hour TTL. Device tokens change when users reinstall apps — cache them with a 24-hour TTL. A cache hit rate of 95% means you serve 19 out of 20 requests from cache instead of hitting the database.
Partitioning notification history prevents table bloat. Partition by month or quarter so each partition contains manageable data volume. Old partitions can be archived to cheaper storage or deleted entirely. Without partitioning, your notification_deliveries table grows to billions of rows and queries slow to a crawl.
Connection pooling prevents database connection exhaustion. Each worker needs database connections, but creating new connections is expensive. Use connection pools that maintain a set of reusable connections. Size pools based on worker count and query concurrency — too small causes contention, too large wastes database resources.
Platform-Specific Implementation Details
Each notification platform has unique quirks and requirements. Understanding these details prevents frustrating bugs and enables you to build more reliable delivery.
Firebase Cloud Messaging (FCM)
FCM handles Android push notifications and also supports iOS and web. It provides both HTTP v1 and legacy APIs, but the legacy API is being phased out. The v1 API requires OAuth 2.0 authentication instead of simple API keys, adding complexity but improving security.
Topic messaging enables broadcast to multiple devices efficiently. Instead of sending to individual tokens, you publish to a topic and FCM fans out to all subscribed devices. This is perfect for global announcements or category-based notifications. Devices subscribe client-side, giving users control over notification categories.
Message priorities affect delivery timing and battery usage. High priority wakes sleeping devices immediately but drains battery faster. Normal priority delivers opportunistically when the device is already awake, saving battery but adding latency. Use high priority sparingly for truly urgent notifications.
Notification and data payloads serve different purposes. Notification payloads display system notifications automatically. Data payloads deliver custom data to your app, which must handle display logic. For maximum flexibility, send both — the system shows a notification, and your app receives data to update its UI.
Error responses indicate why delivery failed. NotRegistered means the token is invalid and should be deleted. InvalidRegistration suggests a malformed token. MessageTooBig indicates payload exceeded the 4KB limit. Handle each error type appropriately to maintain clean token databases and avoid repeated failures.
Apple Push Notification Service (APNs)
APNs requires certificate-based authentication or token-based authentication. Certificate authentication uses a .p12 file that expires annually and must be renewed. Token-based authentication uses a .p8 key that never expires, making it the preferred approach for new implementations.
HTTP/2 connections must be maintained carefully. APNs uses HTTP/2 for all communication, allowing multiple requests over a single connection. However, idle connections get closed after about 30 minutes. Your client library should implement connection pooling and automatic reconnection to avoid failed requests.
Environment selection determines which APNs server to use. Development uses api.sandbox.push.apple.com for apps distributed via Xcode. Production uses api.push.apple.com for App Store apps. Sending to the wrong environment causes cryptic authentication errors.
Critical alerts bypass Do Not Disturb but require special entitlements. These play sound and display even when the device is silenced, suitable for critical safety alerts. Apple requires App Store approval for critical alert entitlement and reserves it for genuinely critical use cases.
Badge numbers must be managed server-side. iOS doesn’t automatically increment badge counts — you must track the count per device and send the current total. When users open your app, send a badge reset request to APNs or send new notifications with badge: 0.
SendGrid and Email Delivery
SendGrid provides both SMTP and HTTP APIs for sending email. The HTTP API is faster and provides more features like template rendering and scheduling. SMTP works with existing email clients but lacks advanced features. For notification systems, use the HTTP API for better control and performance.
Dynamic templates separate content from code. Define email templates in SendGrid’s UI with Handlebars syntax for variable substitution. Your application sends template data via the API without containing HTML in code. This allows non-engineers to update email content without code deployments.
Webhook events provide delivery and engagement data. Configure webhooks to receive events like delivered, opened, clicked, bounced, and spam_report. Process these events to update notification status, remove invalid addresses, and calculate engagement metrics. Use signed webhook requests to verify events are genuinely from SendGrid.
Suppression lists prevent sending to addresses that have bounced or unsubscribed. SendGrid maintains these lists automatically based on delivery feedback. Before sending, check if an address is suppressed. Attempting to send to suppressed addresses wastes credits and damages reputation.
IP reputation affects deliverability significantly. Shared IPs mean your reputation is influenced by other SendGrid customers’ behavior. Dedicated IPs give you full control but require sending volume to maintain reputation. For high-volume senders with good practices, dedicated IPs improve deliverability.
Building Your Notification System: Code Examples
Let’s look at practical code examples for implementing core notification system components. These examples use Python with popular libraries but the patterns apply to any language.
Notification Gateway API
The gateway provides a REST API for submitting notifications. It validates requests, applies rate limiting, and publishes to the message queue. This example uses FastAPI for the web framework and Kafka as the message broker.
Message Queue Consumer
The consumer processes messages from Kafka, applies business logic, and routes to channel-specific queues. It handles errors gracefully with retry logic and dead letter queue support.
Push Notification Delivery Worker
The delivery worker sends push notifications via FCM with proper error handling, retry logic, and token management. It processes messages in batches for efficiency.
Common Pitfalls and How to Avoid Them
After building notification systems for years, I’ve seen the same mistakes repeatedly. Here are the most common pitfalls and how to avoid them.
Not Implementing Idempotency
Mistake: Allowing duplicate notifications when systems retry failed requests. Network issues cause timeouts, but the request might have succeeded. Retrying creates duplicates that confuse and annoy users.
Solution: Include a unique idempotency key with every notification request. Store processed keys in Redis with a 24-hour TTL. Before processing a message, check if its key exists in the cache. If found, skip processing — the notification was already handled. This prevents duplicates even with unlimited retries.
Ignoring User Preferences
Mistake: Sending notifications through channels users have disabled. This happens when preference checking occurs at the wrong layer or is skipped entirely under load.
Solution: Always check preferences before routing to channel queues, never at the channel delivery layer. Cache preferences aggressively to avoid database bottlenecks. Implement preference precedence rules — global opt-out overrides all other settings. Log preference violations for debugging.
Poor Token Management
Mistake: Storing invalid device tokens forever and repeatedly attempting delivery. This wastes resources and damages your standing with push notification services.
Solution: Process delivery feedback from FCM and APNs to identify invalid tokens. Mark them as inactive immediately and exclude from future deliveries. Implement periodic cleanup jobs that purge tokens inactive for more than 90 days. Update tokens when users reinstall apps or change devices.
Synchronous Delivery Blocking Application Code
Mistake: Calling notification APIs synchronously from application code, blocking user-facing requests while waiting for delivery. A slow email service can make your web application feel sluggish.
Solution: Always use asynchronous message queues. Your application publishes to the queue and returns immediately. Background workers handle actual delivery. If the queue is unavailable, fail fast and log the error rather than blocking user requests.
Insufficient Monitoring and Alerting
Mistake: Not monitoring notification delivery metrics until users complain. Silent failures mean users miss critical notifications without anyone knowing.
Solution: Track delivery rate, error rate, queue depth, and processing latency for each channel. Alert when delivery rate drops below threshold or error rate spikes. Monitor dead letter queue depth as an early warning of systematic problems. Implement health checks that send test notifications and verify delivery.
Not Planning for Failure
Mistake: Assuming external services will always be available. FCM, APNs, SendGrid, and Twilio all have outages. Your system needs to handle these gracefully.
Solution: Implement circuit breakers that detect failing services and stop sending requests temporarily. Use exponential backoff for retries. Queue messages during outages and drain the queue when services recover. Have fallback mechanisms — if push fails, fall back to in-app notification.
Production Best Practices
Running notification systems in production requires operational excellence beyond just writing code. These practices ensure reliability, debuggability, and maintainability.
Comprehensive Logging
Log every stage of notification processing with structured logging. Include notification ID, user ID, channel, timestamp, and outcome in every log entry. Use correlation IDs to trace a notification through the entire pipeline from creation to delivery. This makes debugging much easier when users report missing notifications.
Avoid logging sensitive data like message content, user contact information, or authentication tokens. Log metadata about the notification — type, priority, channel — but not the actual message. This protects user privacy while maintaining debuggability.
Metrics and Dashboards
Track key metrics for each notification channel. Delivery rate measures what percentage of notifications reach users successfully. Open rate shows engagement for push and email. Click-through rate indicates how compelling your notifications are. Response time tracks end-to-end latency from event to delivery.
Build dashboards that show these metrics in real-time. Include trend lines to identify degradation early. Alert on anomalies — delivery rate dropping below 95%, error rate exceeding 1%, queue depth growing continuously, or latency exceeding SLAs.
Testing Strategies
Unit tests verify individual components like preference checking, template rendering, and retry logic. Integration tests validate interactions between components — does the processor correctly route messages based on user preferences? End-to-end tests send test notifications through the entire system and verify delivery.
Load testing reveals bottlenecks before production. Simulate traffic spikes to verify your system can handle 10x normal load. Test failure scenarios — what happens when FCM is down for an hour? Does your system recover gracefully when services return?
Gradual Rollouts and Feature Flags
Roll out new notification types gradually. Start with 1% of users, monitor for issues, then increase to 10%, 50%, and finally 100%. Use feature flags to enable or disable notification types without code deployments. If a new notification type causes problems, disable it immediately via feature flag.
A/B test notification content and delivery strategies. Try different message templates, send times, and channel combinations to optimize engagement. Measure open rates and click-through rates to determine what resonates with users.
Documentation and Runbooks
Document your architecture with clear diagrams showing data flow. Maintain runbooks for common operational tasks — how to add a new notification type, how to debug missing notifications, how to handle external service outages. Include troubleshooting guides with common error codes and their solutions.
Keep notification templates and configuration in version control. This provides audit history for changes and enables easy rollbacks. Review notification content with product and legal teams before launching to ensure compliance with regulations and brand guidelines.
Wrapping Up
You now understand how notification systems work at scale, from basic architecture to production challenges. We’ve covered the complete journey of a notification through multiple processing stages, examined platform-specific delivery requirements, and explored solutions to scaling challenges that only emerge at millions of notifications per day.
The key insights to remember are that notification systems require asynchronous architecture with message queues, each delivery channel has unique constraints and best practices, user preferences and notification batching prevent fatigue, and comprehensive monitoring reveals problems before users notice them.
Whether you’re building a notification system from scratch, optimizing an existing one, or preparing for system design interviews, these architectural patterns and operational practices will serve you well. The complexity of notification systems often surprises engineers, but understanding the fundamentals makes them approachable.
What notification challenges are you facing in your systems? Have you encountered platform-specific quirks that surprised you? I’d love to hear about the interesting problems you’ve solved.