[Discussion] Adding a pipeline for this example

Proposal: Add CI/CD Pipeline for this template

Last updated: June 6th

Abstract

This proposal aims to add an additional capability to include a CI/CD Pipeline after an user completes its first deployment, so that it becomes a progressive experience after getting used to the programming and deployment model without having to enforce that a Pipeline should be used from Day 1.

Background

When customers complete the steps outlined in the README upon the project generation they expressed they’d like to progress in their experience and understand how to create a Pipeline that will help them “productionize” their Serverless App, however they may not have experience with all services required to do so.

Proposal

I propose 2 methods to implement this that wouldn’t affect the current experience this template provides:

1. Develop a CI/CD pipeline in a separate Github repo that could be pulled into an existing Python project
2. Include a pipeline option in cookiecutter.json and if chosen buildspec.yaml and pipeline.yaml would be added upon project generation

Here’s how the Developer Experience would look like for method 1:

1. Initialize a new app: sam init --runtime python or sam init --location gh:aws-samples/cookiecutter-aws-sam-python
2. Work on business logic and deploy their service without any pipeline
3. Once they feel comfortable with the environment they can add generate a 3-stage CI/CD pipeline with sam init --location gh:aws-samples/cookiecutter-aws-sam-pipeline
4. This would add 3 files to their current project folder: buildspec.yaml (if not already created), pipeline.yaml and Pipeline-Instructions.MD
   • buildspec.yaml would contain comments on how to create a build step (e.g install deps here, run unit tests her, etc.) including the sam package command that will work for the pipeline
   • pipeline.yaml would contain a Cloudformation Template that has everything they’d need to generate this CI/CD pipeline including comments on how to extend or understand options used
   • Pipeline-Instructions.MD would contain information about
     • How the Pipeline is constructed
     • Any changes that may be required to customize the experience (e.g. Single Account vs Multi-Account)
     • How to create a Cloudformation stack to generate this pipeline
     • How to push their code to the new source code repository

Here’s how the Developer Experience would look like for method 2:

1. Initialize a new app: sam init --runtime python or sam init --location gh:aws-samples/cookiecutter-aws-sam-python
2. Work on business logic and deploy their service without any pipeline
3. buildspec.yaml and pipeline.yaml would be included upon project generation and already customized with defaults for the chosen runtime
4. README.MD would be extended to include details on how to generate this CI/CD pipeline under Appendix

Rationale

Each method has its pros and cons and this aims to outline them to aid with a decision alongside the comments by the community too.

Develop a CI/CD pipeline in a separate Github repo

Pros:

• Clean start and separation as pipeline can have different contributors
• One could fork and create a pipeline for any Runtime with their own defaults in the future with subtle changes
• Can add a pipeline to an existing project without impacting it

Cons:

• One needs to know of this template’s existence otherwise it wouldn’t know how to use it

Include a pipeline option upon project generation

Pros:

• Single repo and easier to correlate target project for this pipeline
• Forking this repository for other runtimes would include the entire package idea (project structure, tests and pipeline)

Cons:

• Customers would need to re-initialize this project only to have access to the pipeline feature

Compatibility

Option 1 doesn’t break compatibility whilst the second would only affect existing customers who already initialized a project with this template.

Implementation

Implementation for either methods would be similar so here are the common parts:

• Create a pipeline.yaml that will implement a 3-stage CI/CD CodePipeline pipeline that will include:
  • CodeCommit or Github as a source code repository
  • CodeBuild for running tests, packaging and any security tests
  • Cloudformation to deploy the SAM stack
  • S3 Bucket for storing Build Artifacts
  • Single or Multi-Account option so an user can have placeholders and an explanation as to how to use that in a multi-account scenario
• Create a buildspec.yaml that will implement the build steps for CodeBuild that will include:
  • Run all tests with a coverage baseline
  • Packaging the Python project with one or more functions
  • Create a SAM package
• Append Pipeline into Appendix to include the following instructions:
  • How to create a pipeline stack
  • How to initialize a repo to connect git changes to this newly created pipeline
  • How to add Pipeline metrics (# of deployments, lead time, etc.)
• Extend hooks to not overwrite any existing buildspec.yaml

For the second approach the following would be needed in addition to the above:

• Extend cookiecutter.json and add a include_pipeline option and default to n
• Extend logic in hooks to clean up buildspec.yaml and pipeline.yaml if include_pipeline isn’t chosen

[heitorlessa]

Published MVP with Single-Account support for CodeCommit and GitHub: sam init --location gh:aws-samples/cookiecutter-aws-sam-pipeline

Let me know what you think and I will work on Multi-account in the coming months

[heitorlessa]

That’s right! It’s a 2-step process:

1. I want to make a PR for SAM CLI to include a buildspec.yaml for each Runtime we support
2. I want to publish a generic CI/CD pipeline in a separate repo