In today's interconnected world, effective communication between applications is crucial. This article delves into the importance of a messaging service for organizations, highlighting our shift from Redis to SQS and addressing the associated challenges.

By the end of this article, readers will have a comprehensive understanding of Amazon SQS, starting from basic concepts to practical applications. We'll also explore the transition of Event Logger from Redis to SQS and the obstacles encountered during the implementation process.

This guide is for those looking for an easily integrated, scalable, and cost-effective messaging service within AWS. It is especially useful for applications transitioning from Redis and seeking a cloud-managed Messaging Queue solution.

Introduction to Amazon SQS.

Amazon SQS, an AWS-managed message queuing service, facilitates asynchronous communication in distributed systems. It allows components to exchange messages, serving as a scalable, reliable, and cost-effective foundation for various cloud-based applications.

Decoupling Components: SQS promotes independence among system components. Producers and consumers can operate separately, enhancing scalability, fault tolerance, and overall architecture flexibility. This decoupling enables the creation of robust and loosely coupled systems.

Queue Types:

• Standard Queues: Offer best-effort ordering and delivery, suitable for scenarios where precise order and exact-once delivery are not critical.
• FIFO (First-In-First-Out) Queues: Guarantee processing of messages exactly once and in the order received. Ideal for applications requiring strict ordering, like financial systems.

Reliability and Durability: SQS ensures message reliability and durability by redundantly storing them across multiple availability zones. This design improves fault tolerance and ensures message preservation even during server or data center issues.

Scalability: SQS is highly scalable, automatically adjusting to handle virtually unlimited messages. This scalability is vital for systems with fluctuating workloads, ensuring SQS can adapt to increased message throughput.

Security Features: SQS provides security through AWS IAM access control, data encryption in transit using HTTPS, and server-side encryption at rest using AWS KMS. These measures protect message confidentiality and integrity.

Visibility Timeout: When a consumer retrieves a message, SQS temporarily makes it invisible to other consumers for a set visibility timeout. This prevents simultaneous processing by multiple consumers, avoiding duplication of work.

Monitoring and Management: AWS offers tools like Amazon CloudWatch metrics for monitoring queue performance and activity. Integration with AWS CloudTrail allows auditing API calls made on SQS resources.

Retention Period: SQS enables configuring the retention period for messages. Unprocessed messages within this period are automatically deleted, managing storage costs.

Integration: SQS seamlessly integrates with various AWS services, playing a crucial role in AWS-based architectures. It commonly pairs with AWS Lambda, Amazon EC2, and other services to build scalable and event-driven applications.

Key Pointers:

• In SQS, messages can be read individually or in batches, with each batch supporting up to 10 messages in a single request. This limit also applies to delete requests.
•  Standard Queue outperforms FIFO Queue significantly in terms of speed.
• To maximize the number of messages in a request, adjust the wait time config (with a maximum wait time of 20 seconds).
• Increase the visibility timeout config (with a maximum of 12 hours) to prevent multiple reads from multiple threads.
• Messages in SQS have a maximum retention period of 14 days; beyond that, they are automatically deleted.
• The maximum size of a message in a queue is 256KB. Messages larger than this size are either uploaded to S3 or can be implemented using message chunking.

Real-World Application: Event-Logger Journey with SQS

In late 2022, Event Logger embarked on its SQS journey to address challenges arising from failed events during outages in writing components like MYSQL, MONGO, REDIS, and KAFKA. 

Choosing SQS as a self-managed solution, Event Logger effectively parked and replayed messages, opting for a FIFO queue to ensure guaranteed order and delivery.

Detailed article : coming soon 🔜

Challenges Faced:

Initially encountering slowness issues when reading messages from the queue, Event Logger transitioned from a FIFO queue to a Standard Queue, resulting in a significant performance improvement. As Event Logger continued to integrate SQS, it replaced Redis in various components to tackle challenges such as server costs, crashes, and the absence of auto-scaling.

Solution:

To address the initial slowness, we switched the queue type from FIFO to Standard, as the FIFO nature was unnecessary. This change led to a remarkable improvement in performance, meeting our expectations.

Ongoing Journey:

Event Logger extended its use of SQS to replace Redis in various use cases. The goal is to eliminate Redis usage entirely, as the challenges faced with Redis, including server costs, crashes, lack of auto-scaling, and the need for manual management, can be mitigated with SQS.

Challenges with Redis:

•  High server costs
•  Crashes due to out-of-memory issues
•  Risk of losing all events in case of server crashes
•  Lack of self-management
•  Lack of auto-scaling
• Pros of SQS:

- Faster performance compared to Redis

Currently, we have transitioned one of our Events (SMS voice fiber) from Redis to SQS, implementing a bridge—a utility Java JAR that facilitates data transfer from Redis to SQS. The EL component then directly reads messages from SQS.

SQS EL Pricing

The provided data illustrates the total number of events processed by EL-fallback-standard-queue-prod-us on January 21, 2023. For details on SQS pricing, please refer to the pricing model [here](link). SQS charges are based on the requests made, including read, write, and delete operations. For each event, we perform three requests (read, write, and delete). SQS supports batching, allowing us to process 10 messages in a single request. The cost is incurred for every 1 million requests at a rate of $0.40. Cost Matrix (without batching for delete):

This means that for 4.98 million data processed on January 21st, the cost is $2.39.

Refer to the SQS Price Chart for January 21st, showing an approximate cost of $2.30.

Enhancement Proposal: Implementing batching for delete requests can significantly reduce costs. The proposed cost with batching for all API requests is as follows:

Total Cost with Batching: 0.5976 .

This implies a 300% cost reduction per day with batching implemented for all API requests.

Kindly utilize the SQS Calculator to obtain an estimate of your SQS usage costs.

Conclusion

In conclusion, our journey through Amazon SQS showcased its vital role in overcoming real-world challenges, as seen in the Event Logger application. The transition from FIFO to Standard Queue and the replacement of Redis underscored SQS's efficiency. 

Exploring the cost analysis, we unveiled a 300% reduction through batching. The SQS Calculator aids in estimating costs. SQS emerges as a cost-effective, scalable solution, crucial for AWS architectures. 

With its user-friendly features and economic advantages, SQS stands as a key player in enhancing communication and system resilience in today's interconnected world.