logging stops after a few hours with "retry total timeout exceeded"

(This is in continuation of #586)

Because of #586 we yanked out the google winston log library and used directly "@google-cloud/logging": "5.3.1",

24 hours went by with no memory leak BUT I just realized that logging did stop about 15 hours after it started. Every attempt to write an entry since then produced the following (thousands of such messages in the log, the one below is the first).

These messages start appearing about 7 hours after the deployment but between the 7th and the 15ht hour I do see some of our log messages as well as the "ERROR: " messages.

The message is written using a console.log() in the error path of the promise. Here is the approximate code that produces this message:

import * as googleLogging from '@google-cloud/logging';
...
const metadata =
        {
            resource: {
                type: "gke_container",
            }
        }
const logger = new googleLogging.Logging().log(
                "name",
            {
                removeCircular: true
            }
        );
const entry = this.logger.entry(metadata, data);
let promise = this.logger.debug(entry);  // this worked fine for several hours
promise.catch(err => {
          // This is what we see after
           console.log(`ERROR: ${util.inspect(err)}`) ;
});

[bcoe]

@gae123 could you please try:

https://www.npmjs.com/package/@google-cloud/logging/v/5.4.1

and let us know if this version is continuing to have problems?

[bcoe]

@gae123 please feel free to reopen this if you bump into this issue again, I think the issues with @grpc/grpc-js we’re currently working on are unrelated to this bug 👍