"File does not contain valid certificates" when trustCerts not specified in TLS client (v1.1.0 only)

When a basic config is provided to use a tls client where trustCerts is not specified, linkerd v1.1.0 responds with HTTP/1.1 502 Bad Gateway when issued the following curl statement: curl -x localhost:4140 -X GET http://www.google.com:443 -sI | head -n 1.

The error log reports File does not contain valid certificates: /tmp/certCollection1848457118942697108.tmp.

When linkerd v1.0.2 is used with the exact same config and curl command, it responds with HTTP/1.1 200 OK.

For reference, here’s the TLS client config:

    kind: io.l5d.static
    configs:
    - prefix: "/$/io.buoyant.rinet/443/{service}"
      tls:
        commonName: "{service}"

Here’s the error log:

E 0621 20:04:51.016 UTC THREAD21 TraceId:d186a04959f5bc12: service failure
java.lang.IllegalArgumentException: File does not contain valid certificates: /tmp/certCollection1848457118942697108.tmp
	at io.netty.handler.ssl.SslContextBuilder.trustManager(SslContextBuilder.java:164)
	at com.twitter.finagle.netty4.ssl.Netty4SslConfigurations$.configureTrust(Netty4SslConfigurations.scala:40)
	at com.twitter.finagle.netty4.ssl.client.Netty4ClientEngineFactory.apply(Netty4ClientEngineFactory.scala:63)
	at com.twitter.finagle.netty3.Netty3Transporter.$anonfun$addFirstTlsHandlers$1(Netty3Transporter.scala:307)
	at com.twitter.finagle.netty3.Netty3Transporter.$anonfun$addFirstTlsHandlers$1$adapted(Netty3Transporter.scala:306)
	at scala.Option.foreach(Option.scala:257)
	at com.twitter.finagle.netty3.Netty3Transporter.addFirstTlsHandlers(Netty3Transporter.scala:306)
	at com.twitter.finagle.netty3.Netty3Transporter.newPipeline(Netty3Transporter.scala:382)
	at com.twitter.finagle.netty3.Netty3Transporter.newConfiguredChannel(Netty3Transporter.scala:391)
	at com.twitter.finagle.netty3.Netty3Transporter.$anonfun$apply$2(Netty3Transporter.scala:398)
	at com.twitter.finagle.netty3.ChannelConnector.apply(Netty3Transporter.scala:48)
	at com.twitter.finagle.netty3.Netty3Transporter.apply(Netty3Transporter.scala:400)
	at com.twitter.finagle.netty3.Netty3Transporter$$anon$3.apply(Netty3Transporter.scala:149)
	at com.twitter.finagle.Http$Client$$anon$2.$anonfun$apply$1(Http.scala:209)
	at com.twitter.util.Local.letClear(Local.scala:151)
	at com.twitter.finagle.context.MarshalledContext.letClearAll(MarshalledContext.scala:112)
	at com.twitter.finagle.context.Contexts$.$anonfun$letClearAll$1(Contexts.scala:38)
	at com.twitter.util.Local.letClear(Local.scala:151)
	at com.twitter.finagle.context.LocalContext.letClearAll(LocalContext.scala:43)
	at com.twitter.finagle.context.Contexts$.letClearAll(Contexts.scala:37)
	at com.twitter.finagle.Http$Client$$anon$2.apply(Http.scala:209)
	at com.twitter.finagle.Filter$AndThen$$anon$3.apply(Filter.scala:149)
	at com.twitter.finagle.pool.CachingPool.apply(CachingPool.scala:58)
	at com.twitter.finagle.pool.WatermarkPool.apply(WatermarkPool.scala:144)
	at com.twitter.finagle.liveness.FailureAccrualFactory.apply(FailureAccrualFactory.scala:370)
	at com.twitter.finagle.service.ExceptionRemoteInfoFactory.apply(ExceptionRemoteInfoFactory.scala:71)
	at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:227)
	at com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:99)
	at com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:99)
	at com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:99)
	at com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:99)
	at com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:99)
	at com.twitter.finagle.loadbalancer.LoadBalancerFactory$StackModule$$anon$2.apply(LoadBalancerFactory.scala:208)
	at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:227)
	at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:227)
	at com.twitter.finagle.loadbalancer.LeastLoaded$Node.apply(LeastLoaded.scala:30)
	at com.twitter.finagle.loadbalancer.Balancer.apply(Balancer.scala:250)
	at com.twitter.finagle.loadbalancer.Balancer.apply$(Balancer.scala:240)
	at com.twitter.finagle.loadbalancer.p2c.P2CLeastLoaded.apply(P2CLeastLoaded.scala:29)
	at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:227)
	at com.twitter.finagle.factory.TrafficDistributor$Distributor.apply(TrafficDistributor.scala:99)
	at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:227)
	at com.twitter.finagle.factory.TrafficDistributor.apply(TrafficDistributor.scala:303)
	at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:227)
	at com.twitter.finagle.factory.StatsFactoryWrapper.apply(StatsFactoryWrapper.scala:44)
	at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:227)
	at com.twitter.finagle.factory.RefcountedFactory.apply(RefcountedFactory.scala:24)
	at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:227)
	at com.twitter.finagle.factory.TimeoutFactory.apply(TimeoutFactory.scala:61)
	at com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:99)
	at com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:99)
	at com.twitter.finagle.service.Retries$$anon$1.applySelf(Retries.scala:225)
	at com.twitter.finagle.service.Retries$$anon$1.apply(Retries.scala:262)
	at com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:99)
	at com.twitter.finagle.service.DelayedFactory.$anonfun$apply$1(DelayedFactory.scala:46)
	at com.twitter.util.Future.$anonfun$flatMap$1(Future.scala:1089)
	at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:107)
	at com.twitter.util.Promise$Transformer.k(Promise.scala:107)
	at com.twitter.util.Promise$Transformer.apply(Promise.scala:117)
	at com.twitter.util.Promise$Transformer.apply(Promise.scala:98)
	at com.twitter.util.Promise$$anon$1.run(Promise.scala:421)
	at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200)
	at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158)
	at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:272)
	at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:108)
	at com.twitter.util.Promise.runq(Promise.scala:406)
	at com.twitter.util.Promise.updateIfEmpty(Promise.scala:801)
	at com.twitter.util.ExecutorServiceFuturePool$$anon$4.run(FuturePool.scala:141)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.cert.CertificateException: found no certificates in input stream
	at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:98)
	at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:64)
	at io.netty.handler.ssl.SslContext.toX509Certificates(SslContext.java:1026)
	at io.netty.handler.ssl.SslContextBuilder.trustManager(SslContextBuilder.java:162)
	... 72 more

[wmorgan]

We’re currently targeting week of July 10th for the next Linkerd release. (Please note this is an estimate, not a guarantee!)

[hawkw]

Hey @gitcarter and @dario-simonetti, just wanted to let you know that we just merged a fix for this issue in commit d879758ee8865bd78703342affd64c935a546299. It should be in the next release.

Thanks for reporting, and let us know if you have any other issues!