Publisher total timeout value change exposing potential long-running publisher issues

Environment details

• OS: Alpine linux
• Node.js version: 14.15
• npm version: 6.14
• @google-cloud/pubsub version: 2.18.1

Issue Summary

After upgrading @google-cloud/pubsub versions from 2.16.6 to 2.18.1 (and not making any other changes), users are seeing a higher rate of occurrence for this error:

"Error: Total timeout of API google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received".

However along with occurring less often on version 2.16.6, the error message was slightly different (the total timeout went from 600000 to 60000):

"Error: Total timeout of API google.pubsub.v1.Publisher exceeded 600000 milliseconds before any response was received".

There was only one change made to the publisher config but it only changed the initial timeout. The publisher config file shows no change in the total timeout between versions 2.18.1 and 2.16.6.

Here is the sample code shared by the user, but it was not changed when the library version was upgraded

    const topicOptions = {
      batching: {
        maxBytes: 5242880,
        maxMessages: 50,
        maxMilliseconds: 400,
      },
    };
const topic = new PubSub(options).topic(topicName, topicOptions);
await topic.publish(Buffer.from(JSON.stringify(normalizedEventData)));

There were also a few recent flaky-bot issues opened with similar error messages:

• https://github.com/googleapis/nodejs-pubsub/issues/1418
• https://github.com/googleapis/nodejs-pubsub/issues/1404

We would like to know whether this would require some tweaks to the way the user is currently publishing, or if this was an unintentional change introduced in the newer versions of the library.

Thanks in advance!

[feywind]

I wanted to repost this on here in case it gets anyone going:

  topic.setPublishOptions({
    gaxOpts: {
      timeout: 600000,
    }
  });

That should theoretically boost the timeout for the gax/rpc calls, back to where it was before the change earlier. I don’t think it solves whatever the underlying issue is, but I’m all about getting people going.

[kylechadha]

@feywind Can confirm the PubSub object is created once and reused for subsequent publishes.