General OpenSslEngine problem at remote address

Issue Type:

• Bug report
• Feature request

What happened: Upgrading to Linkerd 1.4.0 with the same client/server configuration causes the following error:

General OpenSslEngine problem at remote address: /10.16.0.94:4141. Remote Info: Not Available

What you expected to happen: I expected Linkerd 1.4.0 to continue working with the same configuration. HTTP server config example:

      servers:
      - port: 4140
        ip: 0.0.0.0
      client:
        tls:
          commonName: "star.domain.inc.com"

HTTP client config example:

      servers:
      - port: 4141
        ip: 0.0.0.0
        tls:
          certPath: /certs/wildcard-proxycert
          keyPath: /certs/wildcard-proxykey

How to reproduce it (as minimally and precisely as possible):

1. Run Linkerd 1.3.6
2. Configure it with the client and server TLS as mentioned above.
3. Update Linkerd to 1.4.0 (this was tested using the linkerd-zipkin build running in Kubernetes).
4. Send a request to the outgoing proxy and the error above is returned.

Anything else we need to know?: This issue happened to both HTTP and HTTP/2 requests after the update to 1.4.0.

Environment:

• linkerd/namerd version, config files: Linkerd 1.4.0, Namerd 1.3.6, HTTP config example as follows:

    - protocol: http
      label: out-http
      identifier:
        kind: io.l5d.path
        segments: 2
        consume: false
      interpreter:
        kind: io.l5d.mesh
        root: /out-http
        dst: /$/inet/<namerd_dns>/4321
      servers:
      - port: 4140
        ip: 0.0.0.0
      client:
        tls:
          commonName: "star.domain.inc.com"

    - protocol: http
      label: in-http
      identifier:
        kind: io.l5d.path
        segments: 2
        consume: true
      interpreter:
        kind: io.l5d.mesh
        root: /in-http
        dst: /$/inet/<namerd_dns>/4321
      servers:
      - port: 4141
        ip: 0.0.0.0
        tls:
          certPath: /certs/wildcard-proxycert
          keyPath: /certs/wildcard-proxykey

• Platform, version, and config files (Kubernetes, DC/OS, etc): Kubernetes 1.10.2
• Cloud provider or hardware configuration: Google Cloud Platform

[mohsenrezaeithe]

Well, I can’t break it anymore on 1.4.4:

$ kubectl logs po/l5d-bhg2h -n mesh
-XX:+AggressiveOpts -XX:+AlwaysPreTouch -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark -XX:InitialHeapSize=33554432 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=357916672 -XX:MaxTenuringThreshold=6 -XX:OldPLABSize=16 -XX:+PerfDisableSharedMem -XX:+PrintCommandLineFlags -XX:+ScavengeBeforeFullGC -XX:-TieredCompilation -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseStringDeduplication 
Aug 01, 2018 3:24:15 AM com.twitter.finagle.http.HttpMuxer$ $anonfun$new$1
INFO: HttpMuxer[/admin/metrics.json] = com.twitter.finagle.stats.MetricsExporter(<function1>)
Aug 01, 2018 3:24:15 AM com.twitter.finagle.http.HttpMuxer$ $anonfun$new$1
INFO: HttpMuxer[/admin/per_host_metrics.json] = com.twitter.finagle.stats.HostMetricsExporter(<function1>)
I 0801 03:24:16.114 UTC THREAD1: linkerd 1.4.4 (rev=4fafcadc2eca0c4a2c227150c4cce64ecfed87a7) built at 20180714-171546
I 0801 03:24:16.800 UTC THREAD1: Finagle version 18.5.0 (rev=225a244e96935278721ea40a54aa2c2e53412f21) built at 20180508-105457
...

cross-cluster routing with both Linkerds on 1.4.4

$ kubectl exec -it l5d-ghsr7 bash -n mesh
[root@l5d-ghsr7 1.4.4]# curl localhost:4140/default/apilogger/build
{"service_name":"apilogger",...}

so I think we can call this fixed! On to Linkerd 1.4+!

[adleong]

This comment: https://github.com/twitter/finagle/pull/678#issuecomment-405775436 suggests that the Netty upgrade may happen in the next Finagle release.