Introduction

In today’s fast-paced 🌐 digital landscape, timely and actionable notifications are crucial for maintaining user engagement 🎯 and ensuring seamless platform operations. Our Notification Centre is designed specifically for CEE System Notifications to efficiently capture platform-generated notifications, ensuring that critical updates such as campaign suspensions 🚫, credit limitations 💳, and system errors ⚡ reach the intended recipients through multiple communication channels.

This system is highly modular, allowing developers to easily group similar notifications, create new notifications, and attach additional handlers seamlessly using the provided SDK 🛠️.

In this blog, we’ll explore the architecture, key components, and data flow of our Notification Centre, detailing how we built a scalable and extensible system using Golang 🐹, Kafka, AWS Lambda ☁️, Kubernetes 🐳, and Elasticsearch 🔍.

Architecture Overview

The Notification Centre is structured into three primary components:

1. Gateway 🎯: Acts as an entry point for handling push and pull notifications.
2. Core ⚙️: Processes and enriches notifications before publishing them for handlers.
3. Handlers 📡: Deliver notifications to various external channels like Elasticsearch, Email, Slack, and WhatsApp.

🖥️ System Architecture Diagram

Components Breakdown

1. Gateway 🎯

The Gateway module is an AWS Lambda function deployed behind an ALB that serves as the entry point for notification requests. It ensures seamless handling of push and pull notifications while maintaining high availability and fault tolerance.

Technology Used: AWS Lambda, ALB (Application Load Balancer), Kafka

Responsibilities:

• Accepting Notification Requests: The Gateway accepts incoming notification requests via REST APIs and validates them against the configured schema. It supports multiple notification types, ensuring flexibility for future notification types.
• Schema Validation: Validates the incoming request payload to ensure data consistency and correctness. Any invalid payload is rejected with an appropriate error message to prevent erroneous data from entering the system.
• Kafka Publishing: Once validated, the Gateway pushes the notification events to the appropriate Kafka topic. Implements a retry mechanism with exponential backoff to handle transient failures while publishing messages to Kafka.
• Elasticsearch Querying: For pull-type notifications, the Gateway queries Elasticsearch to fetch relevant notification data. Provides paginated responses for efficient data retrieval.
• Error Handling: Graceful error handling ensures appropriate HTTP status codes and error messages are returned to the client.

The Gateway acts as the first line of defence and ensures that only valid and enriched notifications enter the system for processing.

2. Core ⚙️

The Core component processes notifications read from Kafka, enriches them with metadata, and consolidates similar events using a time window. It ensures that notifications are optimized and delivered in a standardized format.

Technology Used: Golang, Kubernetes (EKS), MySQL, Redis, Kafka

Responsibilities:

• Reading from Kafka: The Core consumes notifications from the designated Kafka topic in real-time. It ensures high throughput by processing messages concurrently, leveraging Kubernetes auto-scaling capabilities.
• Metadata Enrichment: Fetches additional metadata from MySQL and Redis to enrich the notification payload with relevant details. This process includes fetching client-specific preferences, notification formatting rules, and relevant business logic.
• Event Grouping and Consolidation: Time-Windowed Grouping: Groups similar notification events based on keys such as cid, notificationid, and mid within a configurable time window. Consolidates multiple events into a single notification to prevent notification overload and reduce noise.
• Anti-Spam and De-duplication: Ensures that duplicate notifications are eliminated by maintaining a deduplication cache. Prevents sending multiple notifications for the same event to the same recipient, thereby improving the user experience.
• Publishing Consolidated Notifications: Publishes enriched and consolidated notifications back to the appropriate Kafka topic for consumption by handlers.
• CRUD APIs for Easy Configuration: Provides APIs to manage notification types, rules, and handlers dynamically. Ensures that new notification types can be added or modified without system downtime.

Important Note: The actual notification data is stored in Elasticsearch, while the metadata related to notifications, such as configuration rules and handlers, is stored in MySQL. The CRUD APIs only modify the metadata and do not alter the actual notifications stored in Elasticsearch.

The Core serves as the brain of the Notification Centre, orchestrating the flow of enriched and optimized notifications to downstream handlers.

3. Handlers 📡

Handlers are responsible for consuming notifications from Kafka and delivering them to the respective external channels without altering the notification format. They provide seamless integration with multiple communication channels.

Technology Used: Elasticsearch, External APIs (Slack, Email, WhatsApp)

Responsibilities:

• Consuming Notifications from Kafka: Handlers subscribe to the Kafka topic that receives consolidated notifications from the Core. They consume messages in parallel, ensuring low-latency notification delivery.
• Channel-Specific Delivery: Each handler is configured to deliver notifications to its designated channel. Supported channels include Elasticsearch for persistence, Slack for real-time alerts, Email for formal communication, and WhatsApp for customer engagement.
• Handler Extensibility: Adding New Handlers: New handlers can be seamlessly integrated by implementing the handler interface and subscribing to the Kafka topic. Handlers can either use the provided SDK or directly subscribe to the Kafka topic for consuming notifications. Plug-and-Play Handlers: The decoupled architecture ensures that new handlers can be added or removed without modifying the Core service. Developers can implement custom handlers for additional channels as needed.
• Error Handling and Retries: Handlers implement retry logic with exponential backoff to account for transient failures. Dead Letter Queues (DLQ) are used to capture unprocessed notifications for further investigation.

Handlers in Scope:

• Elasticsearch 🔍 (Default storage for all notifications)

Future Scope Handlers:

• Email 📧 (SMTP-based email notifications)
• Slack 💬 (Slack webhook-based alerts)
• WhatsApp 📲 (WhatsApp API-based messaging)

Handlers play a crucial role in ensuring that notifications are delivered to the right channels in a reliable and efficient manner.

Data Flow

1. Notification Creation: An event (e.g., campaign suspension) triggers a notification request to the Gateway. The Gateway validates and pushes the request to Kafka.
2. Core Processing: The Core reads the message from Kafka. It fetches metadata from MySQL or Redis. The Core groups, deduplicates, and consolidates notifications before publishing them to Kafka.
3. Handlers Processing: The appropriate handler reads the notification from Kafka. The handler delivers the notification to its target destination (e.g., Elasticsearch, Slack, Email, WhatsApp).

API Specifications

1. Notification Publish API

The Notification Publish API accepts POST requests with event payloads and validates incoming data.

📥 Request Example:

Response Example:

2. CRUD APIs for Configuring Notifications

The CRUD APIs allow seamless management of notification configurations and handlers. These APIs provide a flexible interface to add, modify, or remove notification types, rules, and handlers without system downtime.

Important Note: The CRUD APIs modify only the metadata stored in MySQL. They do not modify actual notifications that are stored in Elasticsearch.

API Endpoints:

• Create Notification Type:
• Endpoint: POST /api/notifications
• Description: Adds a new notification type with custom rules and handlers.
• Request Body Example:

- Response:

Read Notification Configurations:

• Endpoint: GET /api/notifications/{notificationid}
• Description: Retrieves configuration details of a specific notification type.
• Response Example:

Update Notification Type:

• Endpoint: PUT /api/notifications/{notificationid}
• Description: Modifies an existing notification type or handler.
• Request Body Example:

• Response:

Delete Notification Type:

• Endpoint: DELETE /api/notifications/{notificationid}
• Description: Removes a notification type and its associated rules.
• Response:

Key Takeaways

• Scalability 📈: Notification Centre uses Kafka and Kubernetes to handle high volumes of notifications efficiently.
• Extensibility 🧩: New handlers can be added without modifying the core architecture by following the provided SDK.
• Performance Optimization ⚡: Metadata caching in Redis and search efficiency through Elasticsearch.
• Error Handling & Retry Mechanisms 🔁: Ensures resilience through exponential backoff for Kafka and HTTP APIs.
• Cloud-Native Design ☁️: AWS Lambda and Kubernetes for optimized workload management.

Conclusion

The Notification Centre built for Netcore System Notifications is a robust and scalable solution for handling platform notifications, ensuring critical messages reach the right audience on time. By leveraging event-driven architecture, cloud-native technologies, and an extensible SDK, this system provides a reliable framework for handling real-time notifications across various channels.

Stay tuned 📢 for further updates as we introduce new enhancements to make notifications even smarter and more user-friendly! 😎