Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Getting GOAWAY when listening to streams after a while

See original GitHub issue

Hi everyone. We have a server listening over many long lived streams using firebase-admin. We recently bumped our versions of @google-cloud/firestore and firebase-admin to:

    "@google-cloud/firestore": "^2.1.0",
    "firebase-admin": "^8.0.0"

We managed to get rid of the DEADLINE_EXCEEDED we were previously getting but unfortunately we are now getting a new error:

Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY

This one happens on different conditions though… While we could reproduce DEADLINE_EXCEEDED with some load tests this ~seems to happen after a few hours~ always happens after 30 minutes or so (perhaps when the streams being listened to are idle for a while?) after the server process starts. Any idea of what might be the cause, and also, which combination of package versions might get us free from both DEADLINE_EXCEEDED and GOAWAY errors? Because when this happens, restarting the process is the only remedy.

Environment details

OS: Happening locally on macOS 10.14.4 and also on a docker container based on the node:10-alpine image running on a Ubuntu 18.04.2 LTS machine.
Node.js version: v10.15.0 on macOS, v10.xon the Ubuntu machine.
npm version: 6.9.0
@google-cloud/firestore version: ^2.1.0

Steps to reproduce

Start the server (which currently listens to ~80 streams)
Leave it running and wait
After a while (maybe after streams have no activity for some time) the error comes up and the streams “stop working”, i.e, no new events are detected starting from there

Thanks!