Attempt to write logs with no transports

This seems to be the issue reported as part of #381 however is not caused by the same problem and a serious problem that I have to put in monitoring to kill my services when I see these log messages.

This is occurring in GKE and only occurs after the service is running for a period of time.

I’m running @google-cloud/logging version 5.2.0 and google-gax 1.1.5.

The log I’m posting below came from a service that started a 6pm PST then at around 20 minutes past midnight, I get one log message:

"(node:34) UnhandledPromiseRejectionWarning: Error: 
    at Http2CallStream.call.on (/tsg/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:96:45)
    at Http2CallStream.emit (events.js:203:15)
    at process.nextTick (/tsg/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:71:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)
" 

Then

[winston] Attempt to write logs with no transports {"warning":{"name":"UnhandledPromiseRejectionWarning"},"level":"warn","message":"Error: \n    at Http2CallStream.call.on (/tsg/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:96:45)\n    at Http2CallStream.emit (events.js:203:15)\n    at process.nextTick (/tsg/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:71:22)\n    at process._tickCallback (internal/process/next_tick.js:61:11)"}

then I just get repeated messages about writing logs with no transports for a while then it just goes completely silent

[alexander-fenster]

I have a repro. Looking into it…

const winston = require('winston');
const {LoggingWinston} = require('@google-cloud/logging-winston');
const loggingWinston = new LoggingWinston();

// Create a Winston logger that streams to Stackdriver Logging
// Logs will be written to: "projects/YOUR_PROJECT_ID/logs/winston_log"
const logger = winston.createLogger({
  level: 'info',
  transports: [
    loggingWinston,
  ],
});

function doLog(size, count) {
  let string = `[W] This is a long log message. Its length is ${size}K. `;
  while (string.length < size * 1024) {
    string += 'x';
  }
  
  for (let i = 0; i < count; ++i) {
    const text = `[${i}] ${string}`;
    logger.info(text);
  }
}

doLog(1200, 1);
setTimeout(() => { doLog(1, 1000); }, 1000);

With this, after I get an expected unhandled rejection for a huge log message, I start getting Winston’s messages about having no transport:

(node:47459) UnhandledPromiseRejectionWarning: Error: Log entry with size 1.17M exceeds maximum size of 256.0K
    at Http2CallStream.<anonymous> (/Users/fenster/logging-repro/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:96:45)
    at Http2CallStream.emit (events.js:201:15)
    at /Users/fenster/logging-repro/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22
    at processTicksAndRejections (internal/process/task_queues.js:82:9)
(node:47459) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:47459) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
[winston] Attempt to write logs with no transports {"message":"[0] [W] This is a long log message. Its length is 1K. xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","level":"info"}

Some preliminary thoughts:

1. I suspect the long message is still coming from your code, or maybe somehow from Winston. We do have request bundling, but the bundling does not group log messages into one entry, and the error says that an entry size exceeds the limit.

2. I could not reproduce any problem using @google-cloud/logging directly without Winston. The Attempt to write logs with no transports error comes from Winston. Unclear yet if it’s Winston fault or our layer (@google-cloud/logging-winston). I’m looking into this now.

Please hold on…

[yinzara]

The problem with logging to stdout is errors. Because stdout considers each line a different log message, you can never actually read an error with more than one line nor does it get properly handled by StackDriver Errors. You also lose the ability to change logging levels and formats dynamically.

If you just update to the latest version of Winston it has my fix in it (i.e. version 3.3.3 or higher).

If you don’t want to upgrade, the actual solution to this problem for most is to just add the following snippet:

log.on("error", error => {
            if (!log.transports.includes(winstonTransport)) {
              log.add(winstonTransport)
            }
          })

where “log” is the winston log instance and “winstonTransport” is a reference to an instance of the LoggingWinston class from @google-cloud/logging-winston