Prisma looses connection to DB in CI

Bug description

I have a CI workflow (github actions) that starts a postgres instance, performs a healthcheck, migrates the database and then runs tests. Heathcheck and migrating prisma all work correctly, however when my tests try to connect to the database, I get the following error:

Can't reach database server at `localhost`:`5432

Please make sure your database server is running at `localhost`:`5432`.

at ../../../node_modules/@prisma/client/runtime/index.js:36269:21

This is my workflow yaml:

name: CI - Check Branch
on: [push]

env:
  NX_CLOUD_DISTRIBUTED_EXECUTION: true
  NEXT_PUBLIC_API_URI: http://example.com
  DATABASE_URL: postgresql://postgres:postgres@localhost/test?schema=public

jobs:
  check-branch:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:14
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432

    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Derive appropriate SHAs for base and head for `nx affected` commands
        uses: nrwl/nx-set-shas@v2
      - uses: actions/setup-node@v1
        with:
          node-version: '16'

      - name: Install packages
        run: yarn install --frozen-lockfile

      - name: Migrate DB & generate prisma client
        run: yarn prisma migrate deploy

      - name: Start agents
        run: npx nx-cloud start-ci-run

      - name: Build
        run: npx nx affected --target=build --parallel --max-parallel=3

      - name: Lint
        run: npx nx affected --target=lint --parallel --max-parallel=2

      - name: Test
        run: npx nx affected --target=test --parallel --max-parallel=2

      - name: End-to-end
        run: npx nx affected --target=e2e --parallel --max-parallel=2

      - name: Stop agents
        run: npx nx-cloud stop-all-agents

  agents:
    runs-on: ubuntu-latest
    name: Agent 1
    timeout-minutes: 60
    strategy:
      matrix:
        agent: [1, 2, 3]
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v1
        with:
          node-version: '16'
      - run: yarn install --frozen-lockfile
      - name: Start Nx Agent ${{ matrix.agent }}
        run: npx nx-cloud start-agent

Prisma seems to initially be able to connect to the DB because migrations work fine:

yarn prisma migrate deploy
  shell: /usr/bin/bash -e {0}
  env:
    NX_CLOUD_DISTRIBUTED_EXECUTION: true
    NEXT_PUBLIC_API_URI: http://example.com
    DATABASE_URL: ***localhost/test?schema=public
    NX_BASE: 394e7021946d4ccbc6b0ba3edc757e6e44be2d69
    NX_HEAD: ad8d60abe6bec038cf9eec5dbbff6e48b0c99ca7
yarn run v1.22.17
$ /home/runner/work/grp-org/grp-org/node_modules/.bin/prisma migrate deploy
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "test", schema "public" at "localhost:5432"

PostgreSQL database test created at localhost:5432

4 migrations found in prisma/migrations

Applying migration `20211218235949_init`
Applying migration `20211219143151_add_basic_user_fields`
Applying migration `20211222211952_add_refresh_token_version_to_user`
Applying migration `20211222212337_add_default_to_user_token_version`

The following migrations have been applied:

migrations/
  └─ 20211218235949_init/
    └─ migration.sql
  └─ 20211219143151_add_basic_user_fields/
    └─ migration.sql
  └─ 20211222211952_add_refresh_token_version_to_user/
    └─ migration.sql
  └─ 20211222212337_add_default_to_user_token_version/
    └─ migration.sql
      
 All migrations have been successfully applied.
Done in 2.32s.

I’ve tried playing with the postgres varibles (host, username, etc) but nothing has worked. This used to happen once every few runs randomly (I think?), but now none of my tests are passing. Any help would be greatly appreciated!

To note:

• I am using an Nx monorepo in this project. I’ve not tested this on any other applications.
• Tests run fine locally (I am using the dotenv cli to manage the DATABASE_URL, but none of this matters in CI since it is a seperate script that does not get used in the workflow) in a postgres instance that is running in a docker container.

How to reproduce

• Create project with prisma (I am using nestjs in a monorepo for my server)
• Add a CI workflow with a postgres instance and provide the database URL in env
• Add tests that need to access the database connection
• Add a CI command to run migrations & tests
• Connection should be initially established with the container, migrations will be applied but tests cannot connect to the database and will fail.

Expected behavior

Prisma should not loose connection, and the tests should be able to access the database

Prisma information

Prosma schema

generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model User {
  id                  Int      @id @default(autoincrement())
  createdAt           DateTime @default(now())
  updatedAt           DateTime @updatedAt
  email               String   @unique
  firstName           String   @db.VarChar(100)
  lastName            String   @db.VarChar(100)
  passwordHash        String
  refreshTokenVersion Int      @default(0)
}

Environment & setup

• OS: Manjaro
• Database: PostgreSQL
• Node.js version: 16

Prisma Version

prisma                  : 3.6.0
@prisma/client          : 3.8.1
Current platform        : debian-openssl-1.1.x
Query Engine (Node-API) : libquery-engine dc520b92b1ebb2d28dc3161f9f82e875bd35d727 (at node_modules/@prisma/engines/libquery_engine-debian-openssl-1.1.x.so.node)
Migration Engine        : migration-engine-cli dc520b92b1ebb2d28dc3161f9f82e875bd35d727 (at node_modules/@prisma/engines/migration-engine-debian-openssl-1.1.x)
Introspection Engine    : introspection-core dc520b92b1ebb2d28dc3161f9f82e875bd35d727 (at node_modules/@prisma/engines/introspection-engine-debian-openssl-1.1.x)
Format Binary           : prisma-fmt dc520b92b1ebb2d28dc3161f9f82e875bd35d727 (at node_modules/@prisma/engines/prisma-fmt-debian-openssl-1.1.x)
Default Engines Hash    : dc520b92b1ebb2d28dc3161f9f82e875bd35d727
Studio                  : 0.440.0
Done in 1.09s.

[benediktms]

I’ve not heard back with any additional info from the nrwl team, but it seems that this is simply an issue with how Nx Cloud works rather than with Prisma itself. I’ve done some more testing and it simply feels like a race condition somewhere. Sometimes Nx Cloud will correctly find the find the database instance, sometimes it won’t. The only other thing I will try is moving the initial database spin up outside of the Nx Cloud worker instance. I’ll do some more testing (as far as you can test race conditions) and see what I can find.

[benediktms]

I will investigate this a bit more. I’m not sure how exactly the Nx Cloud agents distribute the work, but I also saw that envrionment variables don’t always get found correctly. It feels like the workers are running in seperate containers that all have their own enviroenment so cannot access the database connection, but I have nothing to back that up except my feeling. I will also try to set up a minimal repo to try to reproduce this.