Blue Green Deployments

After some workshops within the team and with some customer feedback we have the following proposal and are seeking feedback from the Lagoon community.

Summary

Lagoon will provide a Blue/Green Deployment system which allows Developers to switch all custom routes from one production environment to another production environment. This allows:

1. 0-Downtime production deployments: Production environment can be deployed, post-rollout tasks ran and tested before actual customers are visiting the site.
2. Fast rollbacks: Switch the routes back to the previous production environment
3. Deployment aborts: never switch the routes to the new production environment

Changes in Lagoon

• Allow two Environments be of type production
• New GraphQL mutation for setting in production environment active while the other one is standby
  • this will also switching Custom Routes from the current active to the new active environment
  • Switching the active production environment will do NOT trigger a redeploy of any environments
• New Drush Site alias with production-active and production-standby (automatically filled with the real environments)
• New routes.production key in .lagoon.yml where custom routes can be defined for the production environment (only these will be switched by the new GraphQL Mutation)

Typical Blue-Green Deployment

Situation

• Two production environments already defined, in this example we call them blue green, each of them is connected to a branch, also called blue and green (same system as today, environment names = branch names)
• .lagoon.yml contains routes:

  routes:
    insecure: Redirect
    active:
      - nginx:
        - www.example.com
        - www.example.net
    standby:
      - nginx:
        - standby.example.com
        - standby.example.net
  

• .lagoon.yml defined post-rollout tasks:

  post-rollout:
  - run:
      name: if this is production, sync from current active production environment
      command: |
          if [[ "$LAGOON_ENVIRONMENT_TYPE" == "production" ]]; then
              drush -y sql-sync @production-active @self
              drush -y rsync @production-active:%files @self:%files
          fi
      service: cli
      shell: bash
  - run:
      name: drush cim
      command: drush -y cim
      service: cli
  - run:
      name: drush updb
      command: drush -y updb
      service: cli
  - run:
      name: drush cr
      command: drush -y cr
      service: cli
  

• we assume that the environment blue was already deployed and is set as the active production environment

Deployment Workflow

1. Developer pushes code update into green branch
2. Lagoon deploys the green environment As green environment is a LAGOON_ENVIRONMENT_TYPE=production environment, drush will run drush -y sql-sync @production-active @self, while production-active is automatically set to the blue enviornment. Therefore syncing the DB and Files from blue to green
3. After deployment finished, Developer tests the green deployment with accessing it via the automatically created routes like: varnish-myproject-green.us.amazee.io, also gives stakeholders possibility to test the site on these URLS (if needed also custom routes for green could be created, like green.example.com)
4. Testing of green deployment is completed and approved
5. Developer triggers a Production Environment switch via GraphQL API or Lagon UI
6. Lagoon triggers: 6.1. Checks all defined routes with a specific annotation routes.active and moves them within Kubernets/OpenShift from the blue namespace to the green namespace 6.1. Checks all defined routes with a specific annotation routes.standby and moves them within Kubernets/OpenShift from the green namespace to the blue namespace 6.2. Lagoon switches internal environment status from: blue=active, green=standby to: green=active, blue=standby 6.3. Lagoon informs Developer about successful switch

Rollback Scenario

This assumes that the deployment need to be switched back.

1. Developer triggers another Production Environment switch:
2. Lagoon Triggers: 3.1. Checks all defined routes under routes.production and moves them within Kubernets/OpenShift from the blue namespace to the green namespace 3.2. Lagoon switches internal environment status from: green=active, blue=standby to: blue=active, green=standby 3.3. Lagoon informs Developer about successful switch

Next Deployment Workflow

Basically same procedure, but instead of pushing into green environment/branch, developer pushes into blue environment/branch

Additional ideas:

• Should we prevent Lagoon from deploying a change into the current active production environment? This could force the developers to always use the blue/green deployment system, but maybe could cause some issues if they want for example push a hotfix into the current active production environment without touching the standby environment
• Technically we could have multiple standby production environments, but this maybe makes it too complex to understand? Is there a use case to have multiple standby production environments?

[seanhamlin]

A few thoughts after reading this thread.

• Having the ability to have a second branch with LAGOON_ENVIRONMENT_TYPE=production sounds like a sensible idea. This way they are treated like both being production (e.g. monitoring, database location, etc)
• I disagree with the description that SQL and file syncing are done automatically upon deploy. IMO ‘refreshing’ from the active production should be done manually always. Perhaps a new UI button to do this?
• On that topic, the Lagoon UI should distinguish which production environment is active (and has the route), and which is standby (and can be promoted)

• Having an environment variable defined in the nginx-php pod of IS_ACTIVE_PRODUCTION (or similar) would be extremely useful - it would also mandate a redeploy of Nginx to apply. This way (for instance) you could tie Shield being being enabled on the standby environment, and not enabled on the active environment. Integration with the environment indicator module would also be possible. This could be considered a security and editor UX issue.
• Standardising on the branch names for blue and green sounds reasonable, else this could get confusing for all if staging and test are the 2 production environments.

[simesy]

okay, cool. Because these use cases cannot be solved by Lagoon, they need knowledge about the application itself, so it is something that the Application needs to handle, not the hosting platform.

Correct, I’m suggesting no change to solution here, just the definition of the problem. Content deployment strategy (one type) is a by product of blue/green.