Integrating AWS SQS and AWS Lambda function with a Node Application — Part 1

What do you do when your application “suddenly” needs to process data that is over 100 times the size of its normal use and cracks under the pressure? When faced with this challenge my initial response was to increase the server resource, unfortunately, this only increased overhead cost and didn’t solve the problem completely.

After some research and analysis of our present system, a cheaper and more effective solution was to decouple our tightly coupled system. However, when you are faced with this kind of problem on your production server with clients on your neck to proffer a solution, you are forced to look for out-of-the-box solutions where you can focus on the implementation itself and worry less about the underlying architecture. This is when you appreciate serverless solutions like the AWS Lambda and Queue services like the AWS Simple Queue Service.

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. AWS SQS allows you to send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.[1]

Picture Amazon SQS as a buffer that receives messages from one part of your application and waits till the function that is meant to process the message is ready before sending the message. This way if there’s a failure, your messages are intact and your application (depending on how it's built) won’t crash. Amazon SQS would keep messages in the queue for a default retention period of 4 days unless you trigger polling of messages. In a tightly coupled system, triggers would be set within the system to poll and process messages. This approach can be seen as problematic as a failure at the point of processing messages would impact users. To solve this you could set up another server and run the function from there, or you could use a serverless solution like the AWS Lambda (which was the solution I picked).

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes. [2]

AWS Lambda and Amazon SQS have a dance-like relationship, such that with a little configuration you can get Lambda to poll messages from the queue, process messages, and delete messages from the queue when they have been successfully processed.

AWS has simplified the process of integrating this system with Node application with the use of their SDK. With a few lines of code, it allows developers to create queues, send messages to the queue, receive messages from the queue, and delete messages from the queue all from a Node application.

Using my application as an example I would attempt to break down the integration in the next part(s) of this article. Highlight issues faced and my solution to those issues. I am open to comments and feedback that could improve my approach and overall results.

References

[1] https://aws.amazon.com/sqs/

[2] https://aws.amazon.com/lambda/

Resources

AWS SDK Guide: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/sqs-examples-using-queues.html

Cover image created by vectorjuice — www.freepik.com