GRPC-JS Error: Metadata key contains illegal characters

Environment details

• OS: Debian (using node:10.15.0 Docker image)
• Node.js version: v10.15.0
• npm version: 6.4.1 (using yarn 1.12.3 though)
• @google-cloud/logging-winston version: 1.1.0

Steps to reproduce

1. Send logs via this logger
2. Wait
3. Get error caught by a reporting library, like Sentry in our case

I’ve been seeing this error coming up sporadically, with different characters in the key each time:

Error: Metadata key "�ˆé" contains illegal characters
  File "app:///../node_modules/@grpc/grpc-js/build/src/client.js", line 101, col 45, in Http2CallStream.call.on
    const error = Object.assign(new Error(status.details), status);
  File "events.js", line 187, col 15, in Http2CallStream.emit
  File "domain.js", line 459, col 23, in Http2CallStream.EventEmitter.emit
  File "app:///../node_modules/@grpc/grpc-js/build/src/call-stream.js", line 71, col 22, in process.nextTick
    this.emit('status', status);
  File "internal/process/next_tick.js", line 61, col 11, in process._tickCallback

It only appeared once we had a deployment with logging using this library, so I’m pretty sure it’s the logging transport, but can’t be 100% as the stack trace starts in the grpc-js dependency.

Since I don’t know where it is coming from, I don’t really know how debug it. I’m guessing because the work is offloaded to the next tick I wouldn’t even be able to catch the library error in my client code either 🤷‍♀

I found this related upstream issue: https://github.com/grpc/grpc-node/issues/519

[alexander-fenster]

Hi @daaain,

We have released google-gax v1.1.5 that pulls @grpc/grpc-js v0.5.2 in which the invalid headers (that are coming from Node.js http2 module) are printed in a warning, but do not throw exception. Please upgrade your dependencies, it should fix the problem for you.

[DominicKramer]

Thank you for opening this issue. I’m transferring the issue to nodejs-logging since interactions with the logging API are not done by this library but are instead done using the nodejs-logging library. As such, I don’t think this issue is specific to the nodejs-logging-winston library.