NullPointerException on Kubernetes 1.6.1

Linkerd Versions: 0.9.0 and 0.9.1 have same error Kubernetes Versions: 1.6.1, 1.6.0

Stack trace on linkerd 0.9.0

E 0417 21:47:41.726 UTC THREAD21: k8s failed to list endpoints
java.lang.NullPointerException
	at io.buoyant.k8s.EndpointsNamer$.io$buoyant$k8s$EndpointsNamer$$getAddrs(EndpointsNamer.scala:116)
	at io.buoyant.k8s.EndpointsNamer$.io$buoyant$k8s$EndpointsNamer$$mkPorts(EndpointsNamer.scala:145)
	at io.buoyant.k8s.EndpointsNamer$NsCache$$anonfun$io$buoyant$k8s$EndpointsNamer$NsCache$$mkSvc$1.apply(EndpointsNamer.scala:246)
	at io.buoyant.k8s.EndpointsNamer$NsCache$$anonfun$io$buoyant$k8s$EndpointsNamer$NsCache$$mkSvc$1.apply(EndpointsNamer.scala:245)
	at scala.Option.map(Option.scala:146)
	at io.buoyant.k8s.EndpointsNamer$NsCache.io$buoyant$k8s$EndpointsNamer$NsCache$$mkSvc(EndpointsNamer.scala:245)
	at io.buoyant.k8s.EndpointsNamer$NsCache$$anonfun$10.apply(EndpointsNamer.scala:226)
	at io.buoyant.k8s.EndpointsNamer$NsCache$$anonfun$10.apply(EndpointsNamer.scala:225)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:252)
	at scala.collection.immutable.List.flatMap(List.scala:344)
	at io.buoyant.k8s.EndpointsNamer$NsCache.initialize(EndpointsNamer.scala:225)
	at io.buoyant.k8s.EndpointsNamer$NsCache.initialize(EndpointsNamer.scala:213)
	at io.buoyant.k8s.Ns$$anonfun$io$buoyant$k8s$Ns$$watch$1$$anonfun$apply$1.apply(Ns.scala:77)
	at io.buoyant.k8s.Ns$$anonfun$io$buoyant$k8s$Ns$$watch$1$$anonfun$apply$1.apply(Ns.scala:76)
	at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$3.apply(Future.scala:1145)
	at com.twitter.util.Try$.apply(Try.scala:15)

Stack trace on 0.9.1

I 0417 21:57:49.256 UTC THREAD1: linkerd 0.9.1
...
E 0417 21:57:52.349 UTC THREAD18: k8s failed to list endpoints
java.lang.NullPointerException
        at io.buoyant.k8s.EndpointsNamer$.io$buoyant$k8s$EndpointsNamer$$getEndpoints(EndpointsNamer.scala:126)
        at io.buoyant.k8s.EndpointsNamer$NsCache$$anonfun$io$buoyant$k8s$EndpointsNamer$NsCache$$mkSvc$1.apply(EndpointsNamer.scala:221
        at io.buoyant.k8s.EndpointsNamer$NsCache$$anonfun$io$buoyant$k8s$EndpointsNamer$NsCache$$mkSvc$1.apply(EndpointsNamer.scala:219
        at scala.Option.map(Option.scala:146)
        at io.buoyant.k8s.EndpointsNamer$NsCache.io$buoyant$k8s$EndpointsNamer$NsCache$$mkSvc(EndpointsNamer.scala:219)
        at io.buoyant.k8s.EndpointsNamer$NsCache$$anonfun$13.apply(EndpointsNamer.scala:200)
        at io.buoyant.k8s.EndpointsNamer$NsCache$$anonfun$13.apply(EndpointsNamer.scala:199)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:252)
        at scala.collection.immutable.List.flatMap(List.scala:344)
        at io.buoyant.k8s.EndpointsNamer$NsCache.initialize(EndpointsNamer.scala:199)
        at io.buoyant.k8s.EndpointsNamer$NsCache.initialize(EndpointsNamer.scala:187)
        at io.buoyant.k8s.Ns$$anonfun$io$buoyant$k8s$Ns$$watch$1$$anonfun$apply$1.apply(Ns.scala:77)
        at io.buoyant.k8s.Ns$$anonfun$io$buoyant$k8s$Ns$$watch$1$$anonfun$apply$1.apply(Ns.scala:76)
        at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$3.apply(Future.scala:1145)
        at com.twitter.util.Try$.apply(Try.scala:15)
        at com.twitter.util.Future$.apply(Future.scala:163)
        at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:1145)
        at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:1144)
        at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:107)
        at com.twitter.util.Promise$Transformer.k(Promise.scala:107)
        at com.twitter.util.Promise$Transformer.apply(Promise.scala:117)
        at com.twitter.util.Promise$Transformer.apply(Promise.scala:98)
        at com.twitter.util.Promise$$anon$1.run(Promise.scala:421)
        at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200)
        at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158)

Full Stack Traces: https://gist.github.com/lkysow/d82d9d59bb6a7776418917560a0c95dd

With these errors Linkerd can’t do any lookups and so no routing, i.e. it is essentially dead.

[wmorgan]

Hi @ahmetb! Thanks for the note. To be clear, the root cause of that incident was a Kubernetes bug. This Linkerd bug compounded the problem, which is bad, but “caused an outage” is an overstatement.

(I’ll also add that this bug has been fixed since Linkerd 1.0, released over 6 months ago.)

[anubhavmishra]

@siggy I see this behaviour for 1.6.0 as well:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T23:37:30Z", GoVersion:"go1.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl -n=kube-system get ep kube-controller-manager -o jsonpath='{.subsets}'
[]

$ kubectl -n=kube-system get ep -o jsonpath='{.items[?(@.metadata.name=="kube-controller-manager")].subsets}'
<nil>