Preface

Welcome to a transformative journey in the world of message routing! This blog is designed for engineers and architects who are eager to elevate their systems communication efficiency and might have explored or used cloud services like AWS Event Bridge. But completely relying on Event Bridge for every event processing may not be a cost-friendly solution. By the end of this article, you will not only understand the intricacies of a Golang-based routing service but also be equipped to implement one.

Imagine transforming your existing routing limitations into a powerhouse of flexibility and reliability. From advanced data transformation to high-performance handling with multiple goroutines, we will explore how to achieve seamless integration and scalability in your projects. Get ready to unlock the full potential of your distributed systems, making them more robust and future-proof!

Prerequisites

To fully benefit from this blog, readers should have a basic understanding of Golang, experience in software development, familiarity with routing concepts, access to a Go development environment, and a curiosity and willingness to learn.

Problem Statement

In many software architectures, integrating and routing messages between different components can be a challenging task. In our existing Java-Redis-Bridge (JRB) service,

we faced limitations in terms of routing capabilities, which had single source and destination reliance on Redis, and lacked support for data transformation. Additionally, error

handling and retry mechanisms were not robust, leading to potential message loss and operational challenges.

Challenges We Faced

1. Single Source and Destination (Redis): Our previous system relied solely on Redis for both sourcing and destination, limiting the scope and scalability of our message routing system.

2. No Support for Data Transformation: Lack of support for data transformation meant that we couldn't adapt our messages to different destination formats, hindering the flexibility of our architecture.

3. Scaling: Scaling was a challenge since it required manual efforts to increase and decrease the JAVA threads based on the load.

Solution

Our journey to a more advanced and flexible message routing system led us to develop a powerful solution in Golang, addressing the shortcomings of the existing JRB.

Architecture

Introduction

For every application utilizing the Message Router, a crucial step is to define a JSON configuration in the format. This configuration will serve as the blueprint for the advanced

routing capabilities we are about to explore. Let's delve into the enhanced features of our Golang-based Message Router.

Support for Multiple Destinations:

Expanding Your Routing Horizons In the ever-evolving landscape of distributed systems, a robust routing service should offer versatility in routing messages to various destinations. Our Golang-based Message Router excels in providing seamless support for a range of destinations, offering you the flexibility needed to adapt to diverse architectural needs. Let's explore the destinations supported by our router:

1. Redis: Leverage the speed and simplicity of Redis as a destination for your routed messages.
2. SQS (Simple Queue Service): Seamlessly integrates with Amazon SQS, allowing you to scale and decouple components within your architecture.
3. Kinesis: Direct your messages to Amazon Kinesis for real-time processing and analytics, ensuring efficient handling of large streams of data.
4. Event Bridge: Connect with AWS EventBridge, enabling event-driven architectures and simplifying the integration of various services within your ecosystem.
5. HTTP Endpoints: Send messages directly to HTTP endpoints, providing a universal and straightforward means of communication with external systems.

This comprehensive support for multiple destinations enhances the adaptability of our Message Router, making it a valuable asset for orchestrating communication across a

diverse array of services. Whether you're working with in-memory databases, cloud-based queuing systems, streaming services, or external APIs, our router ensures that

your messages find their way to the right destination with ease. Advanced Routing Capabilities In our quest to enhance the routing capabilities of our message router service, we've introduced a powerful feature that selects the destination based on input conditions.

This feature allows for dynamic routing decisions by leveraging a JSON structure with two top-level logical operators: 'and' and 'or'.

Interpretation:

• 'and' Block: All conditions inside the 'and' block must be true for the overall 'and' block to be true. For example, the user must be either "Alice" or "Bob," and the action must be either "purchase" or "refund."
• 'or' Block: At least one condition inside the 'or' block must be true for the overall 'or' block to be true. For instance, either the amount must be 50 or 100, or the status must be either "approved" or "pending."

If the entire condition is true then the destination attached to the condition is chosen as the event target.

This condition is evaluated for each of the messages the message router picks to process.

This flexible and dynamic approach empowers users to define complex routing logic based on various input conditions. The system intelligently processes these conditions,

allowing for sophisticated destination selection tailored to specific use cases.

In your configurations, you can adapt the JSON structure to include different logical operators, additional conditions, and varied key-value pairs. This feature provides a

powerful tool for crafting intricate routing rules, ensuring that your message router service can dynamically adapt to diverse scenarios and meet the demands of your unique

use cases.

Advanced On-the-Fly Data Transformation with Golang Templates

One of the standout features of our message router lies in its capability to perform advanced on-the-fly data transformations using Golang templates. Golang templates

provide a powerful and flexible way to manipulate data, allowing for dynamic transformations based on intricate rules and conditions. Let's delve into this feature with some compelling examples:

Example 1: Basic Data Transformation

Suppose we have an incoming message in JSON format:

Now, we want to transform this into a different format for a specific destination, say Kinesis Data Firehose. We can use a Golang template like this:

In this example, the Golang template allows us to dynamically structure the output based on the incoming data.

Example 2: Advanced Conditional Transformation

Let's take a scenario where we want to apply different transformations based on the 'action' field in the incoming data:

This example demonstrates the flexibility of Golang templates in handling conditional transformations based on input parameters.

Example 3: Iterative Transformation

Consider a scenario where the incoming data contains an array of items, and we want to transform each item individually:

We can use Golang templates to iterate over the array:

This showcases the ability to dynamically process and transform arrays within the incoming data.

Input validation

The message also ensures to validation of the input based on the need. It takes the JSON schema as an input in config to validate the input input against the schema, Ensuring the

integrity of incoming data, preventing malformed data from disrupting the routing process.

Advanced Retry and Error Handling with DLQ:

Message router efficiently retries failed events that failed to reach the destination having intelligent mechanisms based on the type of destination.

Failed events are redirected to a dedicated ERROR queue in Redis which is for now the only supported DLQ.

Auto Scale in Kubernetes

Our message router is designed to seamlessly integrate with Kubernetes, ensuring automatic scaling based on workload demands. This feature enhances the system's

reliability and performance, adapting to varying workloads effortlessly.

High Performance with Multiple Goroutines

Message router harnesses the power of concurrent processing in Golang, our router utilizes multiple goroutines for parallel execution, significantly improving performance and

reducing latency.

These go routines are configurable based on the demand and load.

You can have separate go routines for each of the destinations and source thereby giving more flexibility and extending throttling capabilities.

Alerts

Integrated alerting mechanisms keep administrators informed about any anomalies, ensuring proactive identification and resolution of potential issues.

The alerts are sent to Slack with an intelligent look-back mechanism which does not flood the slack with alerts.

BENCHMARKING

CONCLUSION

In conclusion, our Golang-based message router provides an all-encompassing solution for intricate routing needs. By overcoming the limitations of the previous JRB, our

system introduces advanced features, scalability in Kubernetes, high performance, and adaptability to diverse destinations and data transformation requirements. With this

solution, you can achieve a seamless and efficient message routing experience, setting the stage for a more robust and future-proof architecture in your projects.