Backstage kubernetes app stop refreshing the catalog and components from the DB

I have created the Backstage App using the create-app, added a few plugins and github authentication. Then created the docker image and the kuberntes deployments. The kubernetes deployment starts successfully, I’m able to browse the Backstage App, the template catalog and the components created.

After a few hours the Backstage App stops trying to refresh the catalog and the components. I’m able to browse the existing catalog and components up to the last refresh. I’m also able to create new components and they get created on the DB, but they are not listed on the portal.

Expected Behavior

Backstage continues to refresh the catalog from the DB

Current Behavior

After a few hours the Backstage App stops trying to refresh from the catalog and the compornenets from DB.

Possible Solution

A workaround could be to restart the deployment after 2hs.

Steps to Reproduce

I followed the default instructions:

Create App

• https://github.com/backstage/backstage/blob/master/docs/getting-started/create-an-app.md
• https://backstage.io/docs/auth/github/provider
• https://backstage.io/docs/auth/#adding-the-provider-to-the-sign-in-page
• https://github.com/backstage/backstage/blob/master/contrib/docs/tutorials/help-im-behind-a-corporate-proxy.md#config

Plugins

• https://github.com/backstage/backstage/tree/master/plugins/github-actions
• https://roadie.io/backstage/plugins/github-pull-requests/?utm_source=backstage.io&utm_medium=marketplace&utm_campaign=github-pull-requests
• https://roadie.io/backstage/plugins/github-insights/?utm_source=backstage.io&utm_medium=marketplace&utm_campaign=github-insights
• https://github.com/backstage/backstage/tree/master/plugins/graphiql

Docker/Kubernetes

• https://github.com/backstage/backstage/blob/master/docs/deployment/docker.md
• https://github.com/backstage/backstage/blob/master/docs/deployment/k8s.md

As suggested I’m using the postgres container.

Context

This is the backstage log in debug mode.

2021-07-15T01:10:42.908Z backstage info Loaded config from app-config.yaml
2021-07-15T01:10:42.918Z backstage info Created UrlReader predicateMux{readers=azure{host=dev.azure.com,authed=false},bitbucket{host=bitbucket.org,authed=false},github{host=github.dxc.com,authed=true},github{h
ost=github.com,authed=false},gitlab{host=gitlab.com,authed=false},fetch{}
2021-07-15T01:10:43.386Z auth info Configuring provider, github type=plugin
2021-07-15T01:10:43.403Z techdocs info Creating Local publisher for TechDocs type=plugin
2021-07-15T01:10:43.408Z search info Added DefaultCatalogCollator collator for type software-catalog type=plugin
2021-07-15T01:10:43.417Z app info Serving static app content from /app/packages/app/dist type=plugin
2021-07-15T01:10:43.736Z app info Injecting env config into module-backstage.aa336aff.js type=plugin
2021-07-15T01:10:43.749Z backstage info Initializing http server
2021-07-15T01:10:43.752Z backstage info Listening on :32700
2021-07-15T01:10:44.618Z catalog debug Skipped stitching of template:default/automation-template-v2, no changes type=plugin
2021-07-15T01:10:44.630Z catalog debug Skipped stitching of template:default/runbook_base, no changes type=plugin
2021-07-15T01:10:44.637Z catalog debug Skipped stitching of template:default/contact_information, no changes type=plugin
2021-07-15T01:10:44.646Z catalog debug Skipped stitching of template:default/infrastructure_environment, no changes type=plugin
2021-07-15T01:10:44.650Z catalog debug Skipped stitching of template:default/system_monitoring, no changes type=plugin
2021-07-15T01:10:44.653Z catalog debug Skipped stitching of template:default/application_installation_procedures, no changes type=plugin
2021-07-15T01:10:44.677Z catalog debug Skipped stitching of template:default/disaster-recovery, no changes type=plugin
2021-07-15T01:10:44.704Z catalog debug Skipped stitching of template:default/runbook-template-v2, no changes type=plugin
2021-07-15T01:10:46.410Z search info Starting all scheduled search tasks. type=plugin
2021-07-15T01:10:46.412Z search debug Collating documents for software-catalog via DefaultCatalogCollator type=plugin
.
.
.
2021-07-15T07:40:51.066Z search debug Collating documents for software-catalog via DefaultCatalogCollator type=plugin
2021-07-15T07:40:51.129Z backstage info ::ffff:127.0.0.1 - - [15/Jul/2021:07:40:51 +0000] "GET /api/catalog/entities HTTP/1.1" 200 - "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type=in
comingRequest

2021-07-15T07:43:57.908Z catalog debug Skipped stitching of component:default/testbase, no changes type=plugin
2021-07-15T07:43:57.921Z catalog debug Skipped stitching of template:default/automation-template-v2, no changes type=plugin
2021-07-15T07:43:57.952Z catalog debug Skipped stitching of template:default/system_monitoring, no changes type=plugin
2021-07-15T07:43:57.966Z catalog debug Skipped stitching of template:default/contact_information, no changes type=plugin
2021-07-15T07:43:58.158Z catalog debug Skipped stitching of location:default/generated-a46dbbea0cbe03cc548edd58c37e953737a2d1f6, no changes type=plugin
2021-07-15T07:44:05.961Z catalog debug Skipped stitching of template:default/runbook_base, no changes type=plugin
2021-07-15T07:44:05.964Z catalog debug Skipped stitching of template:default/disaster-recovery, no changes type=plugin
2021-07-15T07:44:05.972Z catalog debug Skipped stitching of template:default/application_installation_procedures, no changes type=plugin
2021-07-15T07:44:05.981Z catalog debug Skipped stitching of template:default/runbook-template-v2, no changes type=plugin
2021-07-15T07:44:09.273Z catalog debug Skipped stitching of location:default/generated-1ed150fa2b6e9ffedd9f55ab9dabd58fa5e15b21, no changes type=plugin
2021-07-15T07:44:09.911Z catalog debug Skipped stitching of template:default/infrastructure_environment, no changes type=plugin

This is the last try to refresh from the DB Then, the log only shows this:

2021-07-15T07:50:51.171Z search debug Collating documents for software-catalog via DefaultCatalogCollator type=plugin
2021-07-15T07:50:51.248Z backstage info ::ffff:127.0.0.1 - - [15/Jul/2021:07:50:51 +0000] "GET /api/catalog/entities HTTP/1.1" 200 - "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type=in
comingRequest

Your Environment

NodeJS Version (v12): v14.15.3 Operating System and Version (e.g. Ubuntu 14.04): Linux backstage-dev 5.4.0-1034-azure backend: added generic dockerfile + docker-compose #35~18.04.1-Ubuntu SMP Thu Dec 10 09:13:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Browser Information: Firefox 85.0 (64-bit) Kubernetes 18

[taras]

@nattymachado and I are seeing the same issue with our deployment. We’ll start looking at this now.

[biancoda]

So, like we discussed, I recreated the docker image with sql instead of postgres DB and I left it running all night. The container started around 2021-07-16T20:05:30.154Z and the last DB refresh was around 2021-07-17T04:55:38.665Z There are no errors on the logs, it just stopped refreshing :S and there were no memory/CPU/storage issues.