Error [ERR_STREAM_WRITE_AFTER_END]: write after end

Environment details

• OS: Google Cloud Container-Optimized OS
• Node.js version: 10.17.0
• npm version: 6.12.0
• @google-cloud/pubsub version: 1.x (1.1.5, tried reverting down to 1.0.0, same error, currently using 0.21.1)

Steps to reproduce

Unclear so far but have been seeing it after publishing 10k+ messages to various topics. Will get back with more details when we can but if anyone else is seeing this to confirm this isn’t a config issue let us know.

We are using our own wrapper around pubsub which you can find here: https://github.com/starchup/google-pubsub-wrapper, extremely bare bones

Error stacktrace

E 2019-11-19T16:49:45.701927239Z events.js:174
E 2019-11-19T16:49:45.702039214Z       throw er; // Unhandled 'error' event
E 2019-11-19T16:49:45.702047675Z       ^
E 2019-11-19T16:49:45.702054717Z 
E 2019-11-19T16:49:45.702060707Z Error [ERR_STREAM_WRITE_AFTER_END]: write after end
E 2019-11-19T16:49:45.702068184Z     at writeAfterEnd (_stream_writable.js:248:12)
E 2019-11-19T16:49:45.702074930Z     at ClientHttp2Stream.Writable.write (_stream_writable.js:296:5)
E 2019-11-19T16:49:45.702081282Z     at filterStack.sendMessage.then.message (/node_modules/@grpc/grpc-js/build/src/call-stream.js:372:34)
E 2019-11-19T16:49:45.702088817Z     at process._tickCallback (internal/process/next_tick.js:68:7)
E 2019-11-19T16:49:45.702094878Z Emitted 'error' event at:
E 2019-11-19T16:49:45.702121372Z     at errorOrDestroy (internal/streams/destroy.js:107:12)
E 2019-11-19T16:49:45.702164237Z     at onwriteError (_stream_writable.js:436:5)
E 2019-11-19T16:49:45.702169699Z     at onwrite (_stream_writable.js:461:5)
E 2019-11-19T16:49:45.702174990Z     at /node_modules/async-listener/glue.js:188:31
E 2019-11-19T16:49:45.702180323Z     at process._tickCallback (internal/process/next_tick.js:63:19)

Thanks!

[feywind]

We are actually able to go up to v0.29.1 without this issue happening so it seems possible that v0.30.0 introduced it (or exposed it) - https://github.com/googleapis/nodejs-pubsub/releases/tag/v0.30.0

Maybe one of the changes to grpc extensions & dependencies has something to do with it. Will try to identify more clearly where it gets thrown tomorrow

Yeah, and this happened in 0.30.0, too:

https://github.com/googleapis/nodejs-pubsub/pull/658

We’re looking at just reverting back to the gRPC C++ bindings for now. I’m talking with some folks on Monday before we commit to that. In the meantime, it’s possible to manually switch to them and see if it helps:

const {PubSub} = require('@google-cloud/pubsub');
const grpc = require('grpc');
const pubsub = new PubSub({grpc});

[feywind]

@shufengh It looks like 2.5.0 is the one that pulled in the new gax version explicitly. Maybe give that one a try?

https://github.com/googleapis/nodejs-pubsub/pull/1085