DB [backstage_plugin_tech-insights] has grown to ~27 GB in less then month

We have started to receive cannot execute INSERT in a read-only transaction during login and various cannot execute SELECT FOR UPDATE in a read-only transaction errors in logs. We are using Azure Flexible PostgreSQL server and we have currently release 2021-12-10 merged. On DB Server I see several sessions in “ROLLBACK” state and some iddle select "final_entities".* from "final_entities" where "final_entities"."final_entity" is not null order by "entity_id" asc…unfortunatelly I have not checked the locked objects… Restart of the server helped…

Thanks to @Rugvip who found we have reached the storage limits of our PostgreSQL server and it has switched to read-only mode.

Expected Behavior

We have not expeted the DB would grow so much.

Current Behavior

Grow ~27GB of data per month for backstage_plugin_tech-insights db (the biggest table is facts, around the same size).

Possible Solution

Steps to Reproduce

Merged release 2021-12-10

Context

It was a DEV DB. PROD DB worked well in the same time. But it was a different “flexi” server.

Your Environment

OS:   Ubuntu 20.04
node: v16.10.0
yarn: 1.22.5
cli:  0.10.1 (installed)

Dependencies:
  @backstage/app-defaults                                  0.1.2
  @backstage/backend-common                                0.9.13
  @backstage/catalog-client                                0.5.2
  @backstage/catalog-model                                 0.9.7
  @backstage/cli-common                                    0.1.6
  @backstage/cli                                           0.10.1
  @backstage/config-loader                                 0.8.1
  @backstage/config                                        0.1.11
  @backstage/core-app-api                                  0.2.0
  @backstage/core-components                               0.8.0
  @backstage/core-plugin-api                               0.3.0
  @backstage/dev-utils                                     0.2.14
  @backstage/errors                                        0.1.5
  @backstage/integration-react                             0.1.15
  @backstage/integration                                   0.6.10
  @backstage/plugin-api-docs                               0.6.18
  @backstage/plugin-app-backend                            0.3.19
  @backstage/plugin-auth-backend                           0.5.0
  @backstage/plugin-azure-devops-backend                   0.2.4
  @backstage/plugin-azure-devops-common                    0.1.2
  @backstage/plugin-azure-devops                           0.1.6
  @backstage/plugin-badges-backend                         0.1.13
  @backstage/plugin-badges                                 0.2.16
  @backstage/plugin-catalog-backend                        0.19.1
  @backstage/plugin-catalog-graph                          0.2.3
  @backstage/plugin-catalog-graphql                        0.2.14
  @backstage/plugin-catalog-import                         0.7.5
  @backstage/plugin-catalog-react                          0.6.6
  @backstage/plugin-catalog                                0.7.4
  @backstage/plugin-circleci                               0.2.31
  @backstage/plugin-cloudbuild                             0.2.29
  @backstage/plugin-code-coverage-backend                  0.1.16
  @backstage/plugin-code-coverage                          0.1.19
  @backstage/plugin-cost-insights                          0.11.13
  @backstage/plugin-explore-react                          0.0.8
  @backstage/plugin-explore                                0.3.22
  @backstage/plugin-gcp-projects                           0.3.10
  @backstage/plugin-github-actions                         0.4.26
  @backstage/plugin-graphiql                               0.2.24
  @backstage/plugin-graphql-backend                        0.1.9
  @backstage/plugin-home                                   0.4.7
  @backstage/plugin-jenkins-backend                        0.1.9
  @backstage/plugin-jenkins                                0.5.14
  @backstage/plugin-kafka-backend                          0.2.12
  @backstage/plugin-kafka                                  0.2.22
  @backstage/plugin-kubernetes-backend                     0.4.0
  @backstage/plugin-kubernetes-common                      0.2.0
  @backstage/plugin-kubernetes                             0.5.0
  @backstage/plugin-lighthouse                             0.2.31
  @backstage/plugin-newrelic                               0.3.10
  @backstage/plugin-org                                    0.3.30
  @backstage/plugin-pagerduty                              0.3.19
  @backstage/plugin-proxy-backend                          0.2.14
  @backstage/plugin-rollbar-backend                        0.1.16
  @backstage/plugin-rollbar                                0.3.20
  @backstage/plugin-scaffolder-backend-module-cookiecutter 0.1.5
  @backstage/plugin-scaffolder-backend-module-rails        0.2.0
  @backstage/plugin-scaffolder-backend                     0.15.16
  @backstage/plugin-scaffolder-common                      0.1.1
  @backstage/plugin-scaffolder                             0.11.14
  @backstage/plugin-search-backend-module-elasticsearch    0.0.6
  @backstage/plugin-search-backend-module-pg               0.2.2
  @backstage/plugin-search-backend-node                    0.4.3
  @backstage/plugin-search-backend                         0.2.8
  @backstage/plugin-search                                 0.5.1
  @backstage/plugin-sentry                                 0.3.30
  @backstage/plugin-shortcuts                              0.1.15
  @backstage/plugin-sonarqube                              0.2.9
  @backstage/plugin-tech-insights-backend-module-jsonfc    0.1.2
  @backstage/plugin-tech-insights-backend                  0.1.3
  @backstage/plugin-tech-insights-common                   0.2.0
  @backstage/plugin-tech-insights-node                     0.1.1
  @backstage/plugin-tech-insights                          0.1.1
  @backstage/plugin-tech-radar                             0.4.13
  @backstage/plugin-techdocs-backend                       0.12.0
  @backstage/plugin-techdocs                               0.12.9
  @backstage/plugin-todo-backend                           0.1.15
  @backstage/plugin-todo                                   0.1.16
  @backstage/plugin-user-settings                          0.3.13
  @backstage/search-common                                 0.2.1
  @backstage/techdocs-common                               0.11.0
  @backstage/test-utils                                    0.1.24
  @backstage/theme                                         0.2.14
  @backstage/types                                         0.1.1
  @backstage/version-bridge                                0.1.1

[Xantier]

There are two types of data that tech insights provides.

1. Version numbers, migration path kind, boolean values what one would use in autotunish purposes
2. Graphable values like various metrics, code base test coverage percentages, etc what one would use to display and study historics on how a component has evolved over time.

The former would likely be something that we want to override on tech-insights level to store latest (or snapshots over a wider time range, depending on how these want to be configured), the latter we’d want to keep ‘forever’ and archive on DB level when the burden gets too big.

The functionality for the first ones isn’t there at the moment, the latter wouldn’t need actions on Backstage side. (We have this on our backlog)

[jvilimek]

OK, I have just cleaned up the data: TRUNCATE TABLE facts. Thanks to both of you! 🙏 @Rugvip & @Xantier !