TOC

Overview

This blog will show how to Process a request from SNS using AWS lambda and pass it for further processing via SQS. These integration can be typically used for infrastructure automation.

  1. Lets say you have an event captured in SNS from monitoring system and now want to take action against it.
  2. Depending on the type of alert you will have lambda function triggered, which will place the request to specific Queue (SQS)
  3. Jenkins job subscribed to the particular Queue will consume data and take appropriate action as defined in the job

Problem Statement

For this new setup, a JSON blob (input) is to be published to an SNS topic, processed by a Lambda function, and then the output pushed into an SQS queue.

Input: {“text”: “Hello World”, “amount”: 10}
Expected Output: {“Result”: “ Hello World”}

What is Lambda ?

AWS’s Lambda is a compute service that lets you run code without provisioning or managing servers. AWS Lambda executes your code only when needed and scales automatically, from a few requests per day to thousands per second.

REF: https://docs.aws.amazon.com/lambda/latest/dg/welcome.html

What is SQS ?

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications

REF: https://aws.amazon.com/sqs/

What is SNS ?

Amazon Simple Notification Service (SNS) is a highly available, durable, secure, fully managed pub/sub messaging service that enables you to decouple microservices, distributed systems, and serverless applications. Amazon SNS provides topics for high-throughput, push-based, many-to-many messaging. Using Amazon SNS topics, your publisher systems can fan out messages to a large number of subscriber endpoints for parallel processing.

REF: https://aws.amazon.com/sns/

Setup Details

Prerequisite

  1. Valid AWS Account with IAM roles for creating LAMBDA, SQS and SNS
  2. AWS CLI setup in your local system to run the cloudformation template

    Note

    This setup may incur cost

Launch the Serverless stack

~: git clone https://github.com/AVM-Consulting/declarative-infrastructure.git
################################# Output #################################
Cloning into 'declarative-infrastructure'...
remote: Enumerating objects: 41, done.
remote: Counting objects: 100% (41/41), done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 41 (delta 5), reused 36 (delta 2), pack-reused 0
Unpacking objects: 100% (41/41), done.
################################# Output #################################

cd declarative-infrastructure/lambda

~/declarative-infrastructure/lambda: aws cloudformation create-stack --stack-name serverless-stack --capabilities CAPABILITY_IAM --template-body file://serverless.yaml

################################# Output #################################
~/declarative-infrastructure/lambda: aws cloudformation create-stack --stack-name serverless-stack --capabilities CAPABILITY_IAM --template-body file://serverless.yaml
{
    "StackId": "arn:aws:cloudformation:us-west-2:303882392497:stack/serverless-stack/e7d5d520-a971-11e9-82dc-0af0816e310e"
}
################################# Output #################################

What did the above command do ?

  • Created a SNS TOPIC called Left-Pad-Topic
  • Created a SQS Queue called serverless-stack-SQSQueue
  • Created a Lambda Function left-Pad-Test1 whose function is to:
    • To accept input from SNS as a json in the form of {“text”:“Hello World”,“space”:“10”}
    • Use the second parameter from the input as a space and publish the output to SQS
    • Expected output will be {“text”:“Hello World”,“space”:“10”} (See Screen Cap)

Note

NOTE: Above is just a hypothetical case main motive is to explain the integration of SQS, SNS with Lambda

Screen Caps

Lambda Function
AWSTemplateFormatVersion: 2010-09-09
Description: CloudFormation Template for LeftPads Test
Resources:
  LambdaExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action: 'sts:AssumeRole'
      ManagedPolicyArns:
      - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
      - 'arn:aws:iam::aws:policy/AmazonSQSFullAccess'
  LambdaFunction1:
    Type: 'AWS::Lambda::Function'
    Properties:
      Code:
        ZipFile: |
          from __future__ import print_function
          import json
          import boto3
          print('Loading function')
          def lambda_handler(event, context):
            message = event['Records'][0]['Sns']['Message']
            json_msg = json.loads(message)
            msg = str(message)
            pad = len(msg) + int(json_msg['space'])
            print("Result: " + msg.rjust(pad))
            cl=boto3.client('sqs')
            queue = cl.list_queues(QueueNamePrefix='serverless-stack')
            cl.send_message(QueueUrl=str(queue["QueueUrls"][0]), MessageBody=msg.rjust(pad))
            return msg.rjust(pad)
      Description:  LeftPad Test
      FunctionName: left-Pad-Test1
      Handler: index.lambda_handler
      Role: !GetAtt
      - LambdaExecutionRole
      - Arn
      Runtime: python2.7
      Timeout: '3'
  LambdaPermission1:
    Type: 'AWS::Lambda::Permission'
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !Ref LambdaFunction1
      Principal: sns.amazonaws.com
      SourceArn: !Ref SNSTopic
  LambdaLogGroup1:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: !Sub "/aws/lambda/${LambdaFunction1}"
      RetentionInDays: '7'
  SNSTopic:
    Type: 'AWS::SNS::Topic'
    Properties:
      DisplayName: Left-Pad-Topic
      TopicName: Left-Pad-Topic
      Subscription:
      - Endpoint: !GetAtt
        - LambdaFunction1
        - Arn
        Protocol: lambda
  SQSQueue:
    Type: 'AWS::SQS::Queue'

  AllowLambds2SQSPolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues: [!Ref 'SQSQueue']
      PolicyDocument:
        Version: '2008-10-17'
        Id: PublicationPolicy1
        Statement:
        - Sid: Allow-Lambda-SendMessage
          Effect: Allow
          Principal: "*"
          Action: "sqs:*"
          Resource: "*"
          Condition:
            ArnEquals:
              aws:SourceArn: !Ref 'LambdaFunction1'

LAMBDA1

SNS Topic -> Click Publish Message

SNS1

SNS-Topic -> Sample Input

SNS2

SQS-Topic -> View output

SQS1

SQS2

Clean the setup

~/declarative-infrastructure/lambda: aws cloudformation delete-stack --stack-name serverless-stack

By the way, check out our best AWS deal: https://www.avmconsulting.net/well-architected-review