delivery: Explicit environment in terraform configuration

Background

The current Terraform implementation has two root modules: ops and app. For each application environment, we expect the developer to remove local state and re-run with a different project ID.

Proposal

Switch to a more conventional model, where our reusable modules are in one directory, and our “root modules” are in an environments directory. Each root module maps to a dedicated environment. ops is special because it’s handles environment-spanning infrastructure, such as Artifact Registry.

.
├── modules
│   ├── app
│   └── delivery
└── environments
    ├── ops
    ├── stage
    └── prod

Once this is done, we add a new environment by copying stage or prod and tweaking. (note, we have some special case logic in a Cloud Build stage to prod handoff.)

In setup.sh, we might setup this terraform structure like so:

gsutil mb gs://${OPS_PROJECT}-tf-state
for d in {'ops','stage','prod'}
do
  pushd $d
  terraform init \
    --backend-config "bucket=${OPS_PROJECT}-tf-state"
    --backend-config "prefix=${d}"
  terraform apply
  popd
done

Terraform-based operations we want to run across environments later could use a script like so:

for d in {'ops','stage','prod'}
do
  terraform -chdir=$d output -raw project_number
done

Problems this will solve

• This will simplify our state management, which currently doesn’t have a way to simultaneously track multiple app environments
• This will remove the naming confusion of an app root module containing an application supporting module
• We will have an approach to minimal duplication of terraform config for environments that still allows environment-based special casing

Additional context

This plan was initially outlined during the prototyping of the last refactor: https://github.com/GoogleCloudPlatform/emblem/tree/main/experimental/flex-env-tf#limitations-of-this-approach

[grayside]

Everything that can be done on this issue has been completed. There are several decision records to be added related to terraform structure and usage, but they can be pursued in a follow-up. Thank you!

[ace-n]

This may break/complicate #347 when merged; we should make sure [we have enough time] to deconflict these two changes.

We’ll also need to propagate #418 into this.