question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Environment details

  • OS: node slim via docker running in GKE
  • Node.js version: 13.3.0
  • npm version: 6.3
  • @google-cloud/pubsub version: 1.5.0
  • grpc version: 1.24.2
  • google gax version: 1.14.1

Steps to reproduce

Randomly we just got an error from our connection to pubsub

message: “Error: Stream removed at MessageStream._onEnd (/app/node_modules/@google-cloud/pubsub/build/src/message-stream.js:234:26) at ClientDuplexStream.<anonymous> (/app/node_modules/@google-cloud/pubsub/build/src/message-stream.js:274:43) at Object.onceWrapper (events.js:411:28) at ClientDuplexStream.emit (events.js:317:22) at endReadableNT (_stream_readable.js:1215:12) at processTicksAndRejections (internal/process/task_queues.js:84:21)”

this also happens locally with the simulator

please help

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:3
  • Comments:30

github_iconTop GitHub Comments

7reactions
Gustencommented, Mar 12, 2020

We’ve started to receive this in our production environment as well and it completely halts the services from receiving messages which is quite the annoyance. We are using @google-cloud/pubsub 1.6.0 and we are also using the solution in https://github.com/googleapis/nodejs-pubsub/issues/825 for injecting grpc manually.

We received this at the exact same time across multiple topics and subscriptions and server instances running a GKE cluster.

4reactions
criveracommented, Feb 28, 2020

Yes we see a little spike in CPU (but that could also be from restarting) and yes no log output. It seems its blocking the whole node thread since the “health” endpoint becomes unavailable and k8s restarts the pod

We do a very standard project setup:

const pubSub = new PubSub({
    projectId: env.GCLOUD_PROJECT
})

@google-cloud/pubsub”: “^1.5.0”,
@grpc/grpc-js”: “^0.6.18”, “google-gax”: “^1.14.1”,

running on node 13.

Nothing has changed

I am able to remove the problem when we go back to before the upgrade at pubsub v 1.1.14

Read more comments on GitHub >

github_iconTop Results From Across the Web

Weird error calling my grpc server "Stream removed" #138
I created new server grpc: dotnet new grpc -o GrpcGreeterServer ather dotnet run after invoke BloomRPC And i received: { "error": "2 ...
Read more >
gRPC java spring server "error": "2 UNKNOWN - Stack Overflow
When trying to call a function from the server I am getting an error { "error": "2 UNKNOWN: Stream removed" }. image of...
Read more >
Flight RPC failed with message: Stream removed
I've tried the available functions (read_all(), read_pandas(), read_chunk()), but no matter which one I use, they eventually fail with the same ...
Read more >
[grpc-io] Re: gRPC Stream Removed Exception
If I now ask the client to talk to the server the > gRPC works fine - does not generate a 'Stream removed'...
Read more >
RTCPeerConnection: removestream event - Web APIs | MDN
The RTCPeerConnection API is now track-based, so having zero tracks in the remote stream is equivalent to the remote stream being removed, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found