mTLS Authentication Broken on Node v15.x / Node v14.15.4

Problem description

Using a custom CA to sign mTLS grpc-js connections results in an error on Node v15, this appears to be a change to how Node handles certificate validation in v15. Using the same code on v14.x results in a successful connection.

Reproduction steps

Setup a mTLS authenticated grpc-js connection using Node v15.5.x

rpcCredentials(): grpc.ChannelCredentials {
  const ca = Buffer.from(this._config.ca_certificate);
  const privateKey = Buffer.from(this._config.private_key);
  const certificate = Buffer.from(this._config.certificate);
  return grpc.credentials.createSsl(ca, privateKey, certificate, {
    checkServerIdentity: () => undefined,
  });
}
const rpc = new foo.RPCClient(this.rpcHost(), this.rpcCredentials(), {
  'grpc.max_send_message_length': 2 * Gb,
  'grpc.max_receive_message_length': 2 * Gb,
});

With GRPC_TRACE all and GRPC_VERBOSITY DEBUG enabled:

2021-01-22T16:59:13.964Z | resolving_load_balancer | dns:example.com:31337 IDLE -> IDLE
2021-01-22T16:59:13.965Z | connectivity_state | dns:example.com:31337 IDLE -> IDLE
2021-01-22T16:59:13.965Z | dns_resolver | Resolver constructed for target dns:example.com:31337
2021-01-22T16:59:13.967Z | channel | dns:example.com:31337 createCall [0] method="/foo.RPC/GetVersion", deadline=Infinity
2021-01-22T16:59:13.968Z | call_stream | [0] Sending metadata
2021-01-22T16:59:13.968Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-22T16:59:13.968Z | call_stream | [0] write() called with message of length 0
2021-01-22T16:59:13.969Z | call_stream | [0] end() called
2021-01-22T16:59:13.969Z | call_stream | [0] deferring writing data chunk of length 5
2021-01-22T16:59:13.969Z | dns_resolver | Resolution update requested for target dns:example.com:31337
2021-01-22T16:59:13.970Z | resolving_load_balancer | dns:example.com:31337 IDLE -> CONNECTING
2021-01-22T16:59:13.971Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-22T16:59:13.971Z | connectivity_state | dns:example.com:31337 IDLE -> CONNECTING
2021-01-22T16:59:13.971Z | resolving_load_balancer | dns:example.com:31337 CONNECTING -> CONNECTING
2021-01-22T16:59:13.971Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-22T16:59:13.971Z | connectivity_state | dns:example.com:31337 CONNECTING -> CONNECTING
2021-01-22T16:59:14.020Z | dns_resolver | Resolved addresses for target dns:example.com:31337: [1.2.3.4:31337]
2021-01-22T16:59:14.021Z | pick_first | Connect to address list 1.2.3.4:31337
2021-01-22T16:59:14.022Z | subchannel_refcount | 1.2.3.4:31337 refcount 0 -> 1
2021-01-22T16:59:14.022Z | subchannel_refcount | 1.2.3.4:31337 refcount 1 -> 2
2021-01-22T16:59:14.022Z | pick_first | Start connecting to subchannel with address 1.2.3.4:31337
2021-01-22T16:59:14.022Z | pick_first | IDLE -> CONNECTING
2021-01-22T16:59:14.022Z | resolving_load_balancer | dns:example.com:31337 CONNECTING -> CONNECTING
2021-01-22T16:59:14.023Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-22T16:59:14.023Z | connectivity_state | dns:example.com:31337 CONNECTING -> CONNECTING
2021-01-22T16:59:14.023Z | subchannel | 1.2.3.4:31337 IDLE -> CONNECTING
2021-01-22T16:59:14.024Z | pick_first | CONNECTING -> CONNECTING
2021-01-22T16:59:14.024Z | resolving_load_balancer | dns:example.com:31337 CONNECTING -> CONNECTING
2021-01-22T16:59:14.024Z | channel | Pick result: QUEUE subchannel: undefined status: undefined undefined
2021-01-22T16:59:14.024Z | connectivity_state | dns:example.com:31337 CONNECTING -> CONNECTING
2021-01-22T16:59:14.160Z | subchannel | 1.2.3.4:31337 connection closed with error self signed certificate
2021-01-22T16:59:14.160Z | subchannel | 1.2.3.4:31337 CONNECTING -> TRANSIENT_FAILURE
2021-01-22T16:59:14.160Z | pick_first | CONNECTING -> TRANSIENT_FAILURE
2021-01-22T16:59:14.160Z | resolving_load_balancer | dns:example.com:31337 CONNECTING -> TRANSIENT_FAILURE
2021-01-22T16:59:14.160Z | channel | Pick result: TRANSIENT_FAILURE subchannel: undefined status: 14 No connection established
2021-01-22T16:59:14.160Z | call_stream | [0] cancelWithStatus code: 14 details: "No connection established"
2021-01-22T16:59:14.161Z | call_stream | [0] ended with status: code=14 details="No connection established"
2021-01-22T16:59:14.161Z | connectivity_state | dns:example.com:31337 CONNECTING -> TRANSIENT_FAILURE

Error: 14 UNAVAILABLE: No connection established
    at Object.callErrorFromStatus (/Users/user/git/foobar/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/Users/user/git/foobar/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
    at Object.onReceiveStatus (/Users/user/git/foobar/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:334:141)
    at Object.onReceiveStatus (/Users/user/git/foobar/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:297:181)
    at /Users/user/git/foobar/node_modules/@grpc/grpc-js/build/src/call-stream.js:130:78
    at processTicksAndRejections (node:internal/process/task_queues:76:11) {
  code: 14,
  details: 'No connection established',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

Environment

• MacOS Big Sur
• Node version v15.5.x
• Node installation method: nvm
• grpc/grpc-js@1.2.4

[moloch--]

@murgatroid99 and any that finds this thread, after must more debugging I identified the problem.

In Node versions up to and including v14.15.3 Node will allow a CA to have a blank CN, however Node versions v14.15.4 and later, Node will reject any CA with a blank CN with no other considerations given to the certificate. To fix for v14.15.4 and later simply provide any non-blank CN value, it’s completely arbitrary it’ll take any non-empty value e.g. a. Yup.

[murgatroid99]

grpc-js just delegates all TLS operations to Node’s own TLS module. There isn’t really any abstraction. If the result of the Node issue you filed is that there is an option to set to get the same behavior as in earlier versions, then I can look into setting that automatically, but if the new version of OpenSSL just has different certificate validation behavior, there isn’t much I can do about that.