Integration testing in AWS

Amazon Web Services (AWS) is an exceptionally robust platform for running applications. It offers a plethora of ready-made tools designed to accelerate the development process, allowing developers to focus on innovation rather than infrastructure. The array of managed services provided by AWS significantly reduces the effort required to set up and maintain the necessary infrastructure, streamlining both deployment and maintenance tasks.

However, like any powerful tool, AWS has its own set of advantages and challenges. While managed services undoubtedly ease the burden on developers in terms of infrastructure management and code development, they introduce complexities in other areas, particularly in integration testing.

Integration Testing of AWS proprietary runtime services – Lambda & StepFunctions

Integration testing often involves complex scenarios, particularly when dealing with runtimes that are not publicly accessible, such as AWS Lambda functions or AWS Step Functions. Despite the perception that such code might not require testing, it is crucial to test these services at the integration level in isolation, rather than relying solely on end-to-end (E2E) testing.

Testing AWS Lambda Functions

For AWS Lambda functions, there are a couple of viable options:

• LocalStack: Utilize the Lambda service feature in LocalStack to emulate the AWS environment locally.
• AWS SAM CLI: Start your Lambda functions locally using the AWS Serverless Application Model (SAM) CLI and run tests against them.

Each of the aforementioned tools has its own set of pros and cons. As mentioned in the ‘Testing done right’ article, Localstack, although a widely used tool, is not developed by AWS itself. This can lead to potential drawbacks such as internal bugs and non-compliance with real AWS Lambda behavior. Considering this, I would recommend starting with tools provided directly by AWS, such as the AWS SAM CLI

1. Define a template yaml file which instructs AWS CLI how the Lambda should be constructed

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.9
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

2. Use sam build | sam local start-api commands which under the hood spins up a new docker container with Lambda image on it that mimics API Gateway integration and allow you to use HTTP REST to query lambda.

In case you would like to test your lambda based on the events you should use sam build | sam local start-lambda commands.

3. Invoke your lambda:

• curl -XGET localhost:3000/hello
• aws lambda invoke --function-name Function --endpoint "http://127.0.0.1:3001" response.json | cat

While the second option (aws lambda invoke) is AWS CLI-specific, the first option (curl -XGET) allows you to use any testing tools you prefer for performing integration tests on your local machine

While not an ideal approach (as it requires constructing an AWS SAM configuration template that may not be needed in production), it can yield valuable insights. Some may argue that such testing is meaningless because we’re creating an artificial SAM YAML file that may not closely resemble the API defined directly on AWS. While this is fundamentally true, it’s crucial to remember that in integration testing, our goal is to verify the correctness of integrations, not necessarily validate contracts. Also, we need to remeber that Lambda’s are designed to be a tiny functions which ideally have one business responsibility. This implies the fact that AWS SAM configuration templates should not be that big 🙂

TLDR: Steps you need to take in order to validate lambda in integration tests.

1. Create you lambda function
2. Define AWS SAM config file with defined REST API
3. Start you lambda locally in docker (or docker-compose with their dependencies)
4. Run tests against started lambda

Testing AWS Step Functions

Testing AWS Step Functions poses a greater challenge comparing to the AWS StepFunctions due to heavy reliance on internal AWS services and integrations. AWS delivers a docker image that allow you to run AWS tests on your local computer (documentation). This kind of tests should be considered as unit testing. They are not checking the integration between various components (e.g. if SFN can reach out to lambda or if SFN can send an event to EventBridge). We can indeed try to mock the responses in the aws-stepfunctions-local tool, however it doesn’t make us sure that integrations will work properly on AWS

Testing AWS Step Functions presents a significant challenge compared to other AWS services, primarily due to its heavy reliance on internal AWS services and integrations. AWS provides a Docker image that allows you to execute Step Functions tests locally on your computer (documentation). These tests are akin to unit testing, focusing on functionality within the Step Functions themselves rather than verifying integrations across different components (such as Step Functions interacting with Lambda or triggering events in EventBridge). While tools like aws-stepfunctions-local allow for mocking responses, they do not guarantee that integrations will function correctly in the AWS environment.

Following question remains – how to ensure that whole Orchestration process within SFN will work correctly.

Testing AWS Step Functions in isolation

One approach to testing the functionality of our Step Functions is to deploy them to the AWS environment and conduct tests there. However, this shifts our focus from integration testing to end-to-end (E2E) testing. To revert to our integration testing strategy, we need to isolate the external components of the Step Functions developed by our teams. It’s crucial not to eliminate calls to AWS managed services like S3, EventBridge, SQS, SNS, etc., as our objective is not to validate these services. Instead, our goal is to ensure that:

• AWS Step Functions are correctly configured and deployable on AWS.
• AWS Step Functions perform their integration tasks effectively, independent of downstream service failures impacting test results

Since we can fairly rule out that AWS Services will fail, we should strive to eliminate the SFN tasks that rely on our code components (e.g. Lambda invocation, REST API invocation)

How to stub services:

• AWS Lambda
  • For each lambda implement stubs within original function. When special query parameter will be provided which indicates that test response should be returned (http://localhost?test_response=true) without invoking real business logic
• Inner AWS SFN call
  • For each SFN create a stub function that always succeeds. Whenever inner function will be called we can steer the response to the caller. From the perspective of a caller it doesn’t matter why function succeeded (if it really did its job or if it failed).
  • For each SFN create a stub function that always fails. Deploy these StepFunction to the AWS and use as the inner dependencies for the SFN being tested. Whenever inner function will be called we can steer the response to either fail or succeed. From the perspective of a caller it doesn’t matter why function succeeded (if it really did its job or if it failed).
• Calls to REST APIs
  • Create a stub with wiremock and deploy them as ECS containers. Make sure to substitude the API URL name to the one contains the mocks.

Do not stub direct interactions with services like S3, DynamoDB, or any other AWS managed service, as AWS is responsible for their correctness. Testing should focus on your code’s logic and interactions rather than the AWS services themselves.

Happy Testing!