Bug: It should be possible to use the SQLite database based on a file

Feature Suggestion

The Backstage example app in this repository uses an in-memory SQLite database. This means that if the user changes the state of the database, and restarts the backend, all the changes disappear. This eventually becomes a source of confusion for Backstage contributors.

If the default SQLite connection were based on filesystem, instead of in-memory, the data will persist. And it will still be lightweight than PostgreSQL.

We already discourage the use of example app in production, hence this change does not increase the risk of people using SQLite in production.

It should be possible to switch the in-memory SQLite config to one that is based on a file.

Possible Implementation

Discovering this PR https://github.com/backstage/backstage/pull/5308 from @Fox32, looks like it’s possible to update the app-config.yaml to

backend:
  database:
    client: sqlite3
    connection:
      filename: 'backstage-db.sqlite'

and start using filesystem based SQLite connection!

But unfortunately this does not work for me and throws a “migrations directory is corrupt” error.

Backend failed to start up Error: The migration directory is corrupt, the following files are missing: 20200511113813_init.js, 20200520140700_location_update_log_table.js, 20200527114117_location_update_log_latest_view.js, 20200702153613_entities.js, 20200721115244_location_update_log_latest_deduplicate.js, 20200805163904_location_update_log_duplication_fix.js, 20200807120600_entitySearch.js, 20200809202832_add_bootstrap_location.js, 20200923104503_case_insensitivity.js, 20201005122705_add_entity_full_name.js, 20201006130744_entity_data_column.js, 20201006203131_entity_remove_redundant_columns.js, 20201007201501_index_entity_search.js, 20201019130742_add_relations_table.js, 20201123205611_relations_table_uniq.js, 20201210185851_fk_index.js, 20201230103504_update_log_varchar.js, 20210209121210_locations_fk_index.js, 20210302150147_refresh_state.js, 20210622104022_refresh_state_location_key.js
    at validateMigrationList (/Users/himanshum/workspace/github.com/backstage/backstage/node_modules/knex/lib/migrations/migrate/Migrator.js:575:11)
    at Migrator.latest (/Users/himanshum/workspace/github.com/backstage/backstage/node_modules/knex/lib/migrations/migrate/Migrator.js:70:7)
    at async Function.create (webpack-internal:///../../plugins/scaffolder-backend/src/scaffolder/tasks/DatabaseTaskStore.ts:70:5)
    at async createRouter (webpack-internal:///../../plugins/scaffolder-backend/src/service/router.ts:95:29)
    at async createPlugin (webpack-internal:///./src/plugins/scaffolder.ts:46:10)
    at async main (webpack-internal:///./src/index.ts:121:32)

Context

Why the default should be in-memory SQLite database which does not persist across sessions?

(I don’t have any strong reasons here to be honest. Please comment!)

• A nudge for people to switch to PostgreSQL so that they don’t accidentally deploy sqlite database.
• Speed?

Why the default should be a file based sqlite database, which persists across sessions?

1. No surprise loss of data for new Backstage integrators.

This has been a constant source of confusion on Discord for new people. Quoting @Pike here (in a call with @adamopenweb) - I was told to use SQLite for my new Backstage app saying it’s safe now and easy to switch later. I lost my populated catalog eventually - so it wasn’t safe.

1. Authentication signing key gets lost, forcing user to sign out and sign in for token verification to work every time

While working with verifying a Backstage token in the backend, we found out that a signing key is created by the auth-backend plugin when the first user signs in and the key is stored in the database. After restarting the backend, the user is still logged in, but there is no signing key available in the database. This means the backstage token can no longer be verified without the user signing out and signing in again. This dramatically increases the debugging cycle.

[ckt114]

This works if I set backend.database.connection.directory: /abs/path/to/sqlite/db/folder.

[OrkoHunter]

Makes sense to me @Rugvip - I’ll re-purpose the issue!

I think we could save some people by adding just a warning log in backend saying that the database used is in-memory and will be cleaned when the backend starts.