Finagle exception: “This writer’s reader has been discarded”

Issue Type:

• Bug report

What happened:

I brought up a simple linkerd+namerd configuration using docker-compose, with three containers: linkerd, namerd, and my service. Ran a workload hitting my API about 30 times per second. My service is configured so that each of these API calls results in two requests to linkerd, which is configured to route those back to another API on the same instance of the same service. (My service is effectively recursing via linkerd.)

Despite all my APIs returning 200s, a few times per minute, I got 502: Bad Gateway from linkerd, along with this exception backtrace on the console:

W 0817 15:16:33.020 UTC THREAD24 TraceId:6cf8fbc2d0b9399f: Exception propagated to the default monitor (upstream address: /172.20.0.3:60158, downstream address: /172.20.0.3:4567, label: #/io.l5d.fs/icecube).

com.twitter.io.Reader$ReaderDiscarded: This writer's reader has been discarded
   at com.twitter.finagle.netty4.http.StreamTransports$$anon$1.discard(StreamTransports.scala:70)
   at com.twitter.finagle.http.DelayedReleaseService$$anon$2.discard(DelayedReleaseService.scala:56)
   at com.twitter.finagle.http.codec.HttpServerDispatcher$$anonfun$handle$2.applyOrElse(HttpServerDispatcher.scala:61)
   at com.twitter.finagle.http.codec.HttpServerDispatcher$$anonfun$handle$2.applyOrElse(HttpServerDispatcher.scala:60)
   at com.twitter.util.Promise.raise(Promise.scala:681)
   at com.twitter.util.Future$JoinPromise$$anonfun$2.applyOrElse(Future.scala:254)
   at com.twitter.util.Future$JoinPromise$$anonfun$2.applyOrElse(Future.scala:251)
   at com.twitter.util.Promise.raise(Promise.scala:681)
   at com.twitter.util.Promise.raise(Promise.scala:686)
   at com.twitter.finagle.http.exp.GenSerialServerDispatcher.$anonfun$new$1(ServerDispatcher.scala:125)
   at com.twitter.util.Future.$anonfun$ensure$1(Future.scala:941)
   at com.twitter.util.Future.$anonfun$ensure$1$adapted(Future.scala:941)
   at com.twitter.util.Promise$Monitored.apply(Promise.scala:202)
   at com.twitter.util.Promise$Monitored.apply(Promise.scala:193)
   at com.twitter.util.Promise$$anon$7.run(Promise.scala:530)
   at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200)
   at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158)
   at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:272)
   at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:108)
   at com.twitter.util.Promise.runq(Promise.scala:520)
   at com.twitter.util.Promise.updateIfEmpty(Promise.scala:877)
   at com.twitter.finagle.netty4.transport.ChannelTransport.com$twitter$finagle$netty4$transport$ChannelTransport$$fail(ChannelTransport.scala:95)
   at com.twitter.finagle.netty4.transport.ChannelTransport$$anon$1.channelInactive(ChannelTransport.scala:186)
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)

What you expected to happen:

Since my service returned 200 for all requests, I expected linkerd to do the same.

How to reproduce it (as minimally and precisely as possible):

Use docker-compose to run three containers:

• linkerd
• namerd
• An http service which, when one API is called, invokes two other APIs on the same service via linkerd

Anything else we need to know?: Some background: I’m attempting to use linkerd and namerd to manage shards in an in-memory database service I’m developing. A query request hits my service, which determines that the queried database has two shards. It therefore produces two more API calls to get the shards, and those requests are routed via linkerd to a node that houses each shard. For now, I only have one node running my service, so the shards happen to be on the same node; therefore, linkerd routes the shard requests back to the same host, effectively causing my service to recurse via linkerd.

Environment:

• linkerd and namerd 1.1.3 from Docker Hub
• Windows 10 plus Docker for Windows
• My Thinkpad

My service is written in Java using the Java Spark framework, and Java 8 streams to make the shard requests run in parallel.

My config files, plus linkerd metrics, are in this Discourse article:

https://discourse.linkerd.io/t/finagle-exception-this-writers-reader-has-been-discarded/244

[wmorgan]

Thanks @prdoyle. We’ll dig into this.

[prdoyle]

@hawkw sounds good. I’ll reopen if I see it again. Thanks!