[RFC] SAM Accelerate (Cloud based development)

We are working on adding a new set of features of the SAM CLI (Serverless Application Model Command Line Interface) that allows you to test against your application in AWS cloud in a production like environment, which you can quickly and iteratively build . Using the sam sync --watch capability lets SAM CLI monitor for any changes to AWS resources defined in your application and automatically sync them in your developer account in AWS cloud so you don’t have to run manual deployments. We are also adding the ability to tail aggregated AWS CloudWatch logs and AWS X-Ray traces in real time.

To summarize, we are working on the following features:

1. The ability to quickly synchronize changes between the local development environment and AWS cloud environment
2. The ability to “tail” aggregated logs for all resources in an application
3. The ability to pull X-Ray traces related to your latest test executions

Want to try out these features in Beta? Instructions are here.

As an enhancement we are considering implementing development environments in a SAM application, tied to an AWS profile and AWS CloudFormation stack.

We would love your feedback/thoughts on this feature! Please feel free to add comments in this issue.

1. What are your thoughts on the sam sync --watch command?
2. What are the biggest pain points in moving to AWS cloud based development in your work environment?
3. What other features would you like to see added here?

Features in development

Project synchronization

The sync command will first check that the CloudFormation Stack exists and then run UpdateStack. After this one-off, initial update of the infrastructure, the SAM CLI will start monitoring local files. Specifically, the SAM CLI will monitor the infrastructure definition files such as CloudFormation template; and the source code for API definitions, State Machine templates, Lambda functions and Layers. Whenever the SAM CLI detects a change, it automatically synchronizes the changes with the AWS cloud using the service APIs and notifies the developer through a message in the console.

$ sam sync --watch --stack-name {stack-name}
  Looking for resources needed for deployment: Found!

        Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-abcde
        A different default S3 bucket can be set in samconfig.toml
        Or by specifying --s3-bucket explicitly.
Queued infra sync. Wating for in progress code syncs to complete...
Starting infra sync. 

.
.
.
Stack update succeeded. Sync infra completed.

{'StackId': 'arn:aws:cloudformation:us-east-1:123456789:stack/sam-app/abcde-12345', 'ResponseMetadata': {'RequestId': 'guid', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'guid', 'content-type': 'text/xml', 'content-length': '377', 'date': 'Thu, 09 Sep 2021 20:41:14 GMT'}, 'RetryAttempts': 0}}
Infra sync completed.

Fetching logs

We are improving sam logs to fetch aggregated logs across multiple AWS services for a CloudFormation stack, and --tail option to fetch them in real time.

$ sam logs --stack-name {stack-name} --tail
2021/09/09/[$LATEST]f9a86d8186be41f580de502740063f78 2021-09-09T21:13:04.750000 START RequestId: e524a367-3194-44f0-bf22-5c688129daec Version: $LATEST
2021/09/09/[$LATEST]f9a86d8186be41f580de502740063f78 2021-09-09T21:13:04.753000 END RequestId: e524a367-3194-44f0-bf22-5c688129daec
2021/09/09/[$LATEST]f9a86d8186be41f580de502740063f78 2021-09-09T21:13:04.754000 REPORT RequestId: e524a367-3194-44f0-bf22-5c688129daec 
Duration: 1.46 ms Billed Duration: 2 msMemory Size: 128 MB Max Memory Used: 48 MB Init Duration: 114.21 ms

Traces

Use SAM CLI to pull X-ray traces using sam traces. You can optionally include a trace-id and use the --tail option to get traces in real time. There is also a --include-traces option with sam logs command to aggregate traces with logs.

$ sam traces --trace-id {trace-id} --tail
XRay Event at (1970-01-19T21:07:02.583000) with id ({trace-id}) and duration (0.005s)
 - 0.005s - build2-HelloWorldFunction-V6h40pCqh7Ma [HTTP: 200]
 - 0.001s - build2-HelloWorldFunction-V6h40pCqh7Ma
   - 0.001s - Invocation
   - 0.000s - Overhead

Features in design

Environment configuration

This is a proposed design for sam env which will manage environment configuration on SAM CLI. Developers can define a “dev” environment for the SAM CLI to operate against. A development environment can be in any account that the developer’s local AWS credentials have access to, however SAM CLI will have guardrails against using “prod” environment. The SAM CLI will first attempt to use the AWS credentials and CloudFormation stack active for the project; if no AWS credentials or CloudFormation stack are configured the SAM CLI asks developers to select an AWS profile, and a CloudFormation stack or create a new one.

// Creating first environment
$ sam env create —name dev
Creating new environment...

New environment: dev successfully created. (#ms)

Your environment: dev is now associated with profile: default and project: Test-app-1. To change the associated AWS profile or project, run the command:
sam env edit —name dev
X-Ray and CloudWatch logs have been enabled on the event sources in your project.

//Trying to create a local environment
$ sam env create —name local
Creating a local environment means it will not be associated with any cloud resources. You can still conduct local testing with select resources.
Do you want to create environment: local anyway? yes/No [y/n]:

//Guardrails against creating a prod environment
$ sam env create —name prod
SAM environments are iterative test resources. It is not recommended to create a production environment.

Do you want to create environment: prod anyway? yes/No [y/n]: n

The SAM CLI saves the development environment information in its configuration file in the local project folder:

…
[environment-dev]
AWSProfile = DevProfile
StackName = devDataProcessingStack 
ProjectType = CloudFormation

[ben-elsen]

@mndeveci I opened https://github.com/aws/aws-sam-cli/issues/3454 I tried to be complete, happy to hear if I must provide more info!

[mndeveci]

@ben-elsen The PR for the fix is already open, can you give it a try? https://github.com/aws/aws-sam-cli/pull/3368