Unable to handle IllegalArgumentException on DefaultHttpHeaders class

Expected Behavior

We should be able to catch the exception / error when this situation happens.

Actual Behavior

It seems we cannot capture or handle this error, everytime it happens, the thread appears to become unresponsive, and our business flow (as well as our fallback strategy) become useless.

java.lang.IllegalArgumentException: only ' ' and '\t' are allowed after '\n': org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void br.com.ifood.logistics.latam.fleetwalletrecharge.recharge.entry.lock.assignment.UnlockedRechargeConsumer.listen(org.springframework.messaging.Message<br.com.ifood.logistics.latam.fleetwalletrecharge.recharge.entry.UnlockedRecharge>,org.springframework.kafka.support.Acknowledgment)' threw exception; nested exception is java.lang.IllegalStateException: stuff happened.; nested exception is java.lang.IllegalStateException: stuff happened.
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:2367)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2338)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2299)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2213)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2138)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2020)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1702)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1272)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1264)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1161)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: stuff happened.
	at br.com.ifood.logistics.latam.fleetwalletrecharge.recharge.entry.lock.assignment.UnlockedRechargeConsumer.process$suspendImpl(UnlockedRechargeConsumer.kt:34)
	at br.com.ifood.logistics.latam.fleetwalletrecharge.recharge.entry.lock.assignment.UnlockedRechargeConsumer.process(UnlockedRechargeConsumer.kt)
	at br.com.ifood.logistics.latam.fleetwalletrecharge.recharge.entry.lock.assignment.UnlockedRechargeConsumer.process(UnlockedRechargeConsumer.kt:15)
	at br.com.ifood.logistics.latam.fleetwalletrecharge.common.integration.consumer.MessageConsumer$onMessage$1$1.invokeSuspend(MessageConsumer.kt:45)
	at br.com.ifood.logistics.latam.fleetwalletrecharge.common.integration.consumer.MessageConsumer$onMessage$1$1.invoke(MessageConsumer.kt)
	at br.com.ifood.logistics.latam.fleetwalletrecharge.common.integration.consumer.MessageConsumer$onMessage$1$1.invoke(MessageConsumer.kt)
	at br.com.ifood.logistics.latam.fleetwalletrecharge.common.application.tracing.NewRelicInterceptor$tap$2.invokeSuspend(NewRelicInstrumentationSupport.kt:65)
	at br.com.ifood.logistics.latam.fleetwalletrecharge.common.application.tracing.NewRelicInterceptor$tap$2.invoke(NewRelicInstrumentationSupport.kt)
	at br.com.ifood.logistics.latam.fleetwalletrecharge.common.application.tracing.NewRelicInterceptor$tap$2.invoke(NewRelicInstrumentationSupport.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165)
	at kotlinx.coroutines.BuildersKt.withContext(Unknown Source)
	at br.com.ifood.logistics.latam.fleetwalletrecharge.common.application.tracing.NewRelicInterceptor.tap(NewRelicInstrumentationSupport.kt:64)
	at br.com.ifood.logistics.latam.fleetwalletrecharge.common.integration.consumer.MessageConsumer$onMessage$1.invokeSuspend(MessageConsumer.kt:44)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

	at io.netty.handler.codec.http.DefaultHttpHeaders$HeaderValueConverterAndValidator.validateValueChar(DefaultHttpHeaders.java:502) ~[netty-codec-http-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.http.DefaultHttpHeaders$HeaderValueConverterAndValidator.convertObject(DefaultHttpHeaders.java:453) ~[netty-codec-http-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.http.DefaultHttpHeaders$HeaderValueConverterAndValidator.convertObject(DefaultHttpHeaders.java:444) ~[netty-codec-http-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.DefaultHeaders.addObject(DefaultHeaders.java:327) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.http.DefaultHttpHeaders.add(DefaultHttpHeaders.java:129) ~[netty-codec-http-4.1.65.Final.jar:4.1.65.Final]
	at software.amazon.awssdk.http.nio.netty.internal.RequestAdapter.lambda$null$2(RequestAdapter.java:94) ~[netty-nio-client-2.16.46.jar:?]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1083) ~[?:?]
	at software.amazon.awssdk.http.nio.netty.internal.RequestAdapter.lambda$addHeadersToRequest$3(RequestAdapter.java:94) ~[netty-nio-client-2.16.46.jar:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
	at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
	at software.amazon.awssdk.http.nio.netty.internal.RequestAdapter.addHeadersToRequest(RequestAdapter.java:94) ~[netty-nio-client-2.16.46.jar:?]
	at software.amazon.awssdk.http.nio.netty.internal.RequestAdapter.adapt(RequestAdapter.java:55) ~[netty-nio-client-2.16.46.jar:?]
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequest(NettyRequestExecutor.java:228) ~[netty-nio-client-2.16.46.jar:?]
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:169) ~[netty-nio-client-2.16.46.jar:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) [netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) [netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) [netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:502) [netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164) [netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) [netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.65.Final.jar:4.1.65.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]

Steps to Reproduce

Basically we’re trying to send a header through a Amazon SDK, this header contains the entire stacktrace of a given exception. It appears that the stacktrace has some illegal characters in it, reported by this illegal argument exception.

We have a bunch of mechanisms that are ready to deal with all sorts of exceptions and errors that a S3 call could produce, but for some reason, this exception in specific cannot by catched and it’s killing the thread itself.

I don’t know if it helps, but we’re using the S3AsyncClient of AmazonSDK, the version will be listed below.

Possible Solution

Your Environment

• Reactor version(s) used: io.projectreactor.netty:reactor-netty-http:1.0.7
• Other relevant libraries versions (eg. netty, …):
  • org.springframework.boot:spring-boot-starter-reactor-netty:2.5.0
  • software.amazon.awssdk:netty-nio-client:2.16.46
• JVM version (java -version): 11
• Kotlin version: 1.5.10
• OS and version (eg. uname -a): Ubuntu 20.04.2 LTS

[LeoFuso]

Oh, this is awkward, I’m sorry. I thought they used the same. Thanks for the investigation, tho. I’ll try to replicate this error anyways, see if I can find anything useful and report to them instead. Thanks for your time!

And I’m sorry for not getting back to you sooner, I’m quite busy these days, hehe. Thanks a lot!

[LeoFuso]

Ok, I’ll try. I’ll need to get in touch with the way the S3 SDK is building the request in order to reproduce it.