Bug: Backend fails to start when using a PostgreSQL connection string

Certain providers inject database connection details into an app as a single connection string environment variable. Backstage configuration supports this via the backend.database.connection.connectionString configuration parameter. When testing this with a PostgreSQL database I however get a startup error.

Expected Behavior

Backend starts.

Current Behavior

On startup, the backend throws the following error and then exits:

Backend failed to start up, Error: The migration directory is corrupt, the following files are missing: 20200511113813_init.js, 20200520140700_location_update_log_table.js, 20200527114117_location_update_log_latest_view.js, 20200702153613_entities.js, 20200721115244_location_update_log_latest_deduplicate.js, 20200805163904_location_update_log_duplication_fix.js, 20200807120600_entitySearch.js, 20200809202832_add_bootstrap_location.js, 20200923104503_case_insensitivity.js, 20201005122705_add_entity_full_name.js, 20201006130744_entity_data_column.js, 20201006203131_entity_remove_redundant_columns.js, 20201007201501_index_entity_search.js, 20201019130742_add_relations_table.js, 20201123205611_relations_table_uniq.js, 20201210185851_fk_index.js

My guess is that the connectionString confuses knex as it tries to run migrations intended for different databases on the same database, causing a conflict/confusion.

Possible Solutions

• Parse the connectionString and pass different database connections to knex
• Allow parsing of connection string environment variables in the configuration language
• Accept workaround of manually editing environment variables (potentially requiring downtime for cloud database maintenance)

Steps to Reproduce

1. npx @backstage/create-app
2. Add docker-compose.yml, Dockerfile and .dockerignore (see contents below)
3. Adjust database config in app-config.yaml to the snippet below.
4. docker-compose up
5. Look at console error output from backend

#Dockerfile
FROM node:12-buster

WORKDIR /usr/src/app

COPY package.json yarn.lock lerna.json /usr/src/app/
COPY packages/app/package.json /usr/src/app/packages/app/
COPY packages/backend/package.json /usr/src/app/packages/backend/
RUN yarn install
COPY . /usr/src/app/
RUN yarn tsc
RUN yarn build

CMD ["yarn", "start"]

#docker-compose.yml
version: '3.4'
services:
  backend:
    build: .
    image: backstage
    command: yarn workspace backend start
    ports:
      - '7000:7000'
    links:
      - database
      - lighthouse
    volumes:
      - ./packages/:/usr/src/app/packages
      - ./app-config.yaml:/usr/src/app/app-config.yaml

  app:
    image: backstage
    command: yarn start
    depends_on:
      - backend
    ports:
      - '3000:3000'
    volumes:
      - ./packages/:/usr/src/app/packages
      - ./app-config.yaml:/usr/src/app/app-config.yaml

  lighthouse:
    image: spotify/lighthouse-audit-service:latest
    environment:
      - PGUSER=postgres
      - PGHOST=database
      - PGPASSWORD=secret
      - PGDATABASE=postgres
      - LAS_CORS=true
      - LAS_PORT=3003
    ports:
      - '3003:3003'
    links:
      - database

  database:
    image: postgres
    ports:
      - '5432:5432'
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=secret

#.dockerignore
.git
node_modules
packages/*/node_modules
plugins/*/node_modules
plugins/*/dist

#app-config.yaml snippet
backend:
  database:
    client: pg
    connection:
      connectionString: postgresql://postgres:secret@database:5432/postgres

Context

Cloud database providers often give connection details in a single connectionString environment variable. Furthermore, during maintenance windows they may modify this environment variable automatically to ensure no downtime for the application - if the application uses that specific environment variable. E.g. hardcoding connection details in another format would cause downtime during such a maintenance window.

Your Environment

• NodeJS Version (v12): v12
• Operating System and Version (e.g. Ubuntu 14.04): Debian (node:12-buster image)
• Browser Information: N/A

[andrewthauer]

Oh, I see it right away @erikxiv. It should actually be:

#app-config.yaml snippet
backend:
  database:
    client: pg
    connection: postgresql://postgres:secret@database:5432/postgres

I suspect this isn’t documented, so this ticket can track documentation of the db connection perhaps.

[freben]

Maybe for starters add examples and clarifying text to the config schema file!