Slamming Pub/Sub with publishJSON requests yields google-gax timeout errors

I’ve noticed numerous other related issues, but followed the recommendation here https://github.com/googleapis/nodejs-pubsub/issues/770#issuecomment-544013584 to post a new issue in isolation for posterity, just in case.

Environment details

• OS: Google App Engine Standard
• Node.js version: ^10
• npm version: N/A
• @google-cloud/pubsub version: 1.5.0

Steps to reproduce

We have a utility service, run only occasionally on demand of our data science team, where we are streaming documents from Firestore, and publishing each document received to a Pub/Sub topic in a different GCP project. The first few thousand messages are published without issue, but at some arbitrary point, about 100 seconds into slamming Pub/Sub with these requests, we repeatedly encounter the following error, after which the App Engine instance is unrecoverable.

Error: Retry total timeout exceeded before any response was received 
    at repeat (/srv/node_modules/@google-cloud/pubsub/node_modules/google-gax/build/src/normalCalls/retries.js:65:31) 
    at Timeout.setTimeout [as _onTimeout] (/srv/node_modules/@google-cloud/pubsub/node_modules/google-gax/build/src/normalCalls/retries.js:100:25) 
    at ontimeout (timers.js:436:11) 
    at tryOnTimeout (timers.js:300:5) 
    at listOnTimeout (timers.js:263:5) 
    at Timer.processTimers (timers.js:223:10) code: 4 } } reason: { Error: Retry total timeout exceeded before any response was received 
  at repeat (/srv/node_modules/@google-cloud/pubsub/node_modules/google-gax/build/src/normalCalls/retries.js:65:31) 
  at Timeout.setTimeout [as _onTimeout] (/srv/node_modules/@google-cloud/pubsub/node_modules/google-gax/build/src/normalCalls/retries.js:100:25) 
  at ontimeout (timers.js:436:11) 
  at tryOnTimeout (timers.js:300:5) 
  at listOnTimeout (timers.js:263:5) 
  at Timer.processTimers (timers.js:223:10) code: 4

I have simplified+obfuscated our code below.

'use strict'

const express = require('express')

const { Firestore } = require('@google-cloud/firestore')
const { PubSub } = require('@google-cloud/pubsub')

const firestore = new Firestore()
const pubsub = new PubSub()

const app = express()

app.get("/", (req, res) => {
    const promises = []
    firestore.collection("someCollection").stream().on('data', (doc) => {
        const promise = new Promise(async (resolve, reject) => {

            // ...
            // data validation/sanitation
            // ...

            pubsub.topic('projects/some-project/topics/some-topic').publishJSON({
                fieldA: data.fieldA,
                fieldB: data.fieldB,
                fieldC: data.fieldC
            }, {
                foo: doc.id,
                bar: 'baz'
            }, (error, messageId) => {
                if (error || !messageId) { reject(error) }
                else resolve(messageId)
            })

        })

        promises.push(promise)
    }).on('end', () => {
        console.log("Received all documents.")

        Promise.all(promises)
        .catch((error) => {
            console.error(`Failed to publish one or more messages. ${JSON.stringify(error, null, 4)}`)
        })
        .finally(() => {
            console.log("Finished dumping collection.")
            res.status(200).send('OK').end()
        })
    })
})

// Start the server
const PORT = process.env.PORT || 8080
app.listen(PORT, () => {
    console.log(`App listening on port ${PORT}`)
})

module.exports = app

Additionally, here is our package.json dependency list including dependencies used but not shown in above code.

"@google-cloud/firestore": "3.5.0",
"@google-cloud/pubsub": "1.5.0",
"@google-cloud/storage": "4.3.1",
"express": "4.17.1",
"lodash": "4.17.15"

Please let me know if I can provide further information useful for finding the problem+solution. Thanks!

[feywind]

It kind of sounds like everyone who was having problems has moved off to other solutions. That’s a bummer, but also I’m glad you’re not blocked! If anyone still ends up needing help or has more comments on it, please feel free to re-open.

[willbattel]

We’re not currently using the program that produced the errors, so I cannot comment on any improvements unfortunately. If we do restore that service I’ll take a look.

In any case, thanks for looking into this!