RFC: Integration tests mechanism

Is this related to an existing feature request or issue?

https://github.com/awslabs/aws-lambda-powertools-python/issues/1009

Which AWS Lambda Powertools utility does this relate to?

Other

Summary

This RFC formalizes our thought process on integration tests (integ test). It also clarifies when functional, integration, and end-to-end tests are best suited.

Requirements

• Setup/tear down overall process should be faster than 2m for 80% of test resources
• Optionally setup/tear down in an independent step to ease step through debugging of multiple independent tests
• Use local codebase while targeting Cloud resources
• No Lambda functions involved
• Ability to delete orphan resources
• Use the same framework to compose and deploy/delete infra as E2E

For security, integ tests will run on either scenarios: 1/ Locally targeting one’s AWS Test Account, 2/ New and Existing PRs upon two approvers, and 3/ Existing PRs labelled run-integ after considered safe by maintainers.

Use case

Integ tests can increase quality confidence in Powertools utilities with direct integration with AWS, e.g. Parameters, Feature Flags, etc.

Functional tests will remain useful to validate interfaces and pure Python logic. However, they can’t help us detect regressions in contracts built on assumptions that integrations would remain backwards compatible.

NOTE: The majority of our tests will continue to be functional tests (subset of integ test) given speed and confidence ratio. Integ tests for extra confidence in direct integrations, and E2E for the complete picture we won’t be able to simulate in either tests.

Proposal

Firstly, we formalize the definition of our tests. There will always be exceptions to the rule and we defer judgement to maintainers, as these can always be reverted if proven inefficient.

Test classification

Test	When to write	Notes	Speed
Unit tests	Verify the smallest possible unit works.	Networking access is prohibited. Prefer Functional tests given our complexity.	Lightning fast (nsec to ms)
Functional tests	Guarantee functionality works as expected. It’s a subset of integration test covering multiple units.	No external dependency. Prefer Fake implementations (in-memory) over Mocks and Stubs.	Fast (ms to few seconds at worst)
Integration tests	Gain confidence that code works with one or more external dependencies.	No need for a Lambda function. Use our code base against an external dependency e.g., fetch an existing SSM parameter.	Moderate to slow (a few minutes)
End-to-end tests	Gain confidence that a Lambda function with our code operates as expected.	It simulates how customers configure, deploy, and run their Lambda function - Event Source configuration, IAM permissions, etc.	Slow (minutes)
Performance tests	Ensure critical operations won’t increase latency and costs to customers.	CI arbitrary hardware can make it flaky. We’ll resume writing perf test after our new Integ/End have significant coverage.	Fast to moderate (a few seconds to a few minutes)

Approach

We will follow the same approach as for E2E - Here’s the most up to date structure.

Out of scope

This RFC doesn’t cover E2E tests, nor contribution guide updates on which tests we will require.

Potential challenges

These are the immediate foreseen challenges:

• Lack of good AWS testing tools. We’ll likely need to create, improve, or contribute to emerging solutions to reduce boilerplate so we can focus on testing contracts only.
• Event data source classes. At the time of writing, there isn’t an official schema for AWS Lambda Events. This means we might reconsider this being out of scope until we have a scalable solution. One idea is to spin up infrastructure on a schedule to generate events for every event data source classes we support. This would be unrelated to our integ test infra.
• Future non-Serverless resources. In the medium-term future, we will support additional Idempotency persistency store like Redis, similarly for Parameters and Feature Flags. This will increase complexity in authoring, maybe maintaining, the CloudFormation approach. We’ll reconsider this approach when we cross that bridge.

Dependencies and Integrations

AWS CloudFormation to create resources required for each feature we want to integ test within a utility. Each utility will dictate which integration will be required.

Services we currently integrate regardless (excluding event sources):

• AWS X-Ray
• Amazon CloudWatch Logs
• Amazon CloudWatch Metrics
• Amazon API Gateway REST API
• Amazon API Gateway HTTP API
• Amazon Elastic Load Balancer
• AWS AppSync
• AWS Lambda
• AWS Systems Manager Parameter Store
• AWS Secrets Manager
• AWS AppConfig
• Amazon DynamoDB
• Amazon DynamoDB Streams
• Amazon SQS
• Amazon SNS
• Amazon Kinesis Data Streams

Alternative solutions

We looked into AWS SDK for speed. However, given the number of integ test and the potential AWS cost it could incur customers in the event of stateful orphan resources (e.g., DynamoDB w/ several records), this became sub-optimal. We could create a static method to list/delete orphan resources for each resource we wanted to test, but this would need more thought on keeping state locally or created resources.

Terraform (TF) was a strong candidate given the speed and state management. Because E2E tests will be done in CloudFormation, it’s more appropriate to stick with the same tool despite CDK usage in E2E tests. On a minor front, TF would need additional tools and docs for maintainers and contributors unfamiliar with it. We can easily reconsider this option, especially if we can achieve 50% greater speed.

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[heitorlessa]

Finally finished the E2E refactoring work. We can now use Integ tests with the same framework. Test definitions have been updated too - I’ll update the RFC body as soon as I get a chance:

Test	When to write	Notes	Speed
Unit tests	Verify the smallest possible unit works.	Networking access is prohibited. Prefer Functional tests given our complexity.	Lightning fast (nsec to ms)
Functional tests	Guarantee functionality works as expected. It’s a subset of integration test covering multiple units.	No external dependency. Prefer Fake implementations (in-memory) over Mocks and Stubs.	Fast (ms to few seconds at worst)
Integration tests	Gain confidence that code works with one or more external dependencies.	No need for a Lambda function. Use our code base against an external dependency e.g., fetch an existing SSM parameter.	Moderate to slow (a few minutes)
End-to-end tests	Gain confidence that a Lambda function with our code operates as expected.	It simulates how customers configure, deploy, and run their Lambda function - Event Source configuration, IAM permissions, etc.	Slow (minutes)
Performance tests	Ensure critical operations won’t increase latency and costs to customers.	CI arbitrary hardware can make it flaky. We’ll resume writing perf test after our new Integ/End have significant coverage.	Fast to moderate (a few seconds to a few minutes)

[pcolazurdo]

Recommendation so we don’t forget: The generated cloudformation should go through a cfn-lint/cfn-guard process before executing it.