Stream: Removed

Environment details

• OS: node slim via docker running in GKE
• Node.js version: 13.3.0
• npm version: 6.3
• @google-cloud/pubsub version: 1.5.0
• grpc version: 1.24.2
• google gax version: 1.14.1

Steps to reproduce

Randomly we just got an error from our connection to pubsub

message: “Error: Stream removed at MessageStream._onEnd (/app/node_modules/@google-cloud/pubsub/build/src/message-stream.js:234:26) at ClientDuplexStream.<anonymous> (/app/node_modules/@google-cloud/pubsub/build/src/message-stream.js:274:43) at Object.onceWrapper (events.js:411:28) at ClientDuplexStream.emit (events.js:317:22) at endReadableNT (_stream_readable.js:1215:12) at processTicksAndRejections (internal/process/task_queues.js:84:21)”

this also happens locally with the simulator

please help

[Gusten]

We’ve started to receive this in our production environment as well and it completely halts the services from receiving messages which is quite the annoyance. We are using @google-cloud/pubsub 1.6.0 and we are also using the solution in https://github.com/googleapis/nodejs-pubsub/issues/825 for injecting grpc manually.

We received this at the exact same time across multiple topics and subscriptions and server instances running a GKE cluster.

[crivera]

Yes we see a little spike in CPU (but that could also be from restarting) and yes no log output. It seems its blocking the whole node thread since the “health” endpoint becomes unavailable and k8s restarts the pod

We do a very standard project setup:

const pubSub = new PubSub({
    projectId: env.GCLOUD_PROJECT
})

“@google-cloud/pubsub”: “^1.5.0”,
“@grpc/grpc-js”: “^0.6.18”, “google-gax”: “^1.14.1”,

running on node 13.

Nothing has changed

I am able to remove the problem when we go back to before the upgrade at pubsub v 1.1.14