NX_* Environment Variables not working on prod builds in our test/staging deployments from gitlab, but only sometimes

Current Behavior

We use environment variables to point to different aws resources in our dev, test, and prod environments.

Locally, we use a .env file, and prefix all of the environment variables with NX_, like we are supposed to, and it all works fine.

In a Dev environment deployment that we set up that is built in a gitlab pipeline, and deployed to an S3 bucket, it also works fine with the env variables we set up in our gitlab CI/CD settings that are scoped to the dev environments.

However, in the exact same kind of pipeline for our Test (Staging) environment, it seems as if the NX build just simply won’t recognize our environmental variables from our gitlab CI/CD settings that we have set there, but sometimes it does. We’ve tried adding the --skip-nx-cache to see if that was messing it up or not. Lately it seems like if we commit and push directly to our main branch, the build will include our env variables just fine, but if we make a merge request, and have a merge commit that results in the pipeline, none of our env variables get used and they are all undefined.

Here are our gitlab-ci steps for both of those pipelines. In the pipeline, we echo some of the variables that have been set as a sanity check to make sure gitlab itself is not misconfigured during the pipeline.

# store node_modules in a cache
cache:
  untracked: true
  key: '*******_portals_node_cache'
  paths:
    - node_modules/

# build apps dev
build_ap_dev:
  stage: build
  dependencies: []
  environment:
    name: ap_dev
    action: prepare
  only:
    - develop
    - /(^\d)*-feature-.*/
    - /(^\d)*-bugfix-.*/
  script:
    - echo "cognito user pool - $NX_COGNITO_USER_POOLS_ID" # successfully echos here, so gitlab variable not a problem
    - yarn build-ap --prod --skip-nx-cache
    - echo "Build Successful"
  artifacts:
    expire_in: 4 hour
    paths:
      - dist/apps/agent-portal-frontend

build_ap_test:
  stage: build
  dependencies: []
  environment:
    name: ap_test
    action: prepare
  only:
    - main
    - /(^\d)*-hotfix-.*/
  script:
    - echo "cognito user pool - $NX_COGNITO_USER_POOLS_ID" # successfully echos here, so gitlab variable not a problem
    - yarn build-ap --prod --skip-nx-cache
    - echo "Build Successful"
  artifacts:
    expire_in: 4 hour
    paths:
      - dist/apps/agent-portal-frontend

package.json build scripts are kept simple, the pipeline adds the flags we need above

Other things from googling around, we’ve tried making sure our environment.prod.ts is like this:

export const environment = {
  production: false,
};

Might this be related to some kind of Nx caching issue that gets stored in our node_modules cache for our pipelines?

Expected Behavior

Nx environment variables should work in all environments (dev, test, prod) when all of them have the same build command.

yarn build-appname --prod

Steps to Reproduce

1. Set up environment variables on gitlab CI/CD that are scoped to specific environments
2. Develop locally with a .env file in the root of the monorepo with variables that are prefixed with NX_*
3. Make Merge Request to develop branch and merge - DEV environment build works fine.
4. Make Merge Request from develop to main branch and merge - TEST environment build is missing env variables.
5. Panic and directly commit some console.logs of env variables on the main branch and run a new pipeline in TEST env - It sometimes magically works, but sometimes not.

Failure Logs

When we download the artifacts from the build to look to see if the variables got used by Nx, we don’t see them in a grep, is this normal?

Environment

env for my local machine, but problem happens in our pipelines that run on a node:15.14 docker image

nx report

>  NX  Report complete - copy this into the issue template

  Node : 15.14.0
  OS   : linux x64
  yarn : 1.22.5
  
  nx : Not Found
  @nrwl/angular : Not Found
  @nrwl/cli : 12.6.4
  @nrwl/cypress : 12.6.4
  @nrwl/devkit : 12.6.4
  @nrwl/eslint-plugin-nx : 12.6.4
  @nrwl/express : Not Found
  @nrwl/jest : 12.6.4
  @nrwl/linter : 12.6.4
  @nrwl/nest : Not Found
  @nrwl/next : Not Found
  @nrwl/node : Not Found
  @nrwl/nx-cloud : Not Found
  @nrwl/react : 12.6.4
  @nrwl/schematics : Not Found
  @nrwl/tao : 12.6.4
  @nrwl/web : 12.6.4
  @nrwl/workspace : 12.6.4
  @nrwl/storybook : 12.7.2
  @nrwl/gatsby : Not Found
  typescript : 4.3.5

[KevinVandy]

Nx why have you forsaken us?

[sohailykhan94]

I was able to get this working using runtimeCacheInputs something like:

"tasksRunnerOptions": {
    "default": {
      "runner": "@nrwl/workspace/tasks-runners/default",
      "options": {
        "cacheableOperations": [
          "build",
          "lint",
          "test",
          "e2e"
        ],
        "runtimeCacheInputs": [
          "echo $NX_ENV_VARIABLE",
          "echo $NX_ENV_VARIABLE_1",
          "echo $NX_ENV_VARIABLE_2",
          "echo $NX_ENV_VARIABLE_3"
        ]
      }
    }
  },

Maybe this might help someone else