Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

reactor actuator invalid response when too much actuator request on decorate-on-each set true

See original GitHub issue

Describe the bug Please provide details of the problem, including the version of Spring Cloud that you are using.

set: webflux + sleuth + actuator version: set(‘springCloudVersion’, “Hoxton.SR1”) and id ‘org.springframework.boot’ version ‘2.2.4.RELEASE’

In decorate-on-each: true setting, actuator not work after 100 time’s call. but, In decorate-on-each: false setting, then, actuator work after 100 time’s call.

this log about In decorate-on-each: true setting, after 100 time’s call, and call /actuator/health then error log

2020-02-11 14:53:50.628 ERROR [,8baf0e4c1ad5671a,8baf0e4c1ad5671a,false] 40736 --- [ctor-http-nio-7] a.w.r.e.AbstractErrorWebExceptionHandler : [da7e3300]  500 Server Error for HTTP GET "/actuator/health"

reactor.core.Exceptions$ReactorRejectedExecutionException: hard cap on deferred tasks reached for boundedElastic("boundedElastic",maxThreads=2,maxTaskQueued=2,ttl=60s)
	at reactor.core.Exceptions.failWithRejected(Exceptions.java:275) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ Handler org.springframework.boot.actuate.endpoint.web.reactive.AbstractWebFluxEndpointHandlerMapping$ReadOperationHandler#handle(ServerWebExchange) [DispatcherHandler]
	|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.cloud.sleuth.instrument.web.TraceWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ HTTP GET "/actuator/health" [ExceptionHandlingWebHandler]
Stack trace:
		at reactor.core.Exceptions.failWithRejected(Exceptions.java:275) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.scheduler.BoundedElasticScheduler.schedule(BoundedElasticScheduler.java:242) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.scheduler.Schedulers$CachedScheduler.schedule(Schedulers.java:1040) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoSubscribeOnCallable.subscribe(MonoSubscribeOnCallable.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:418) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:363) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:418) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:243) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:363) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2199) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:103) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1637) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1637) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1637) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1637) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:247) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:329) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2199) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:132) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:318) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:145) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:274) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:851) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:96) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2199) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.request(ScopePassingSpanSubscriber.java:79) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2007) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:1881) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onSubscribe(ScopePassingSpanSubscriber.java:71) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:441) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at org.springframework.cloud.sleuth.instrument.web.TraceWebFilter$MonoWebFilterTrace.subscribe(TraceWebFilter.java:180) ~[spring-cloud-sleuth-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4105) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.2.RELEASE.jar:3.3.2.RELEASE]
		at reactor.netty.http.server.HttpServerHandle.onStateChange(HttpServerHandle.java:64) ~[reactor-netty-0.9.4.RELEASE.jar:0.9.4.RELEASE]
		at reactor.netty.tcp.TcpServerBind$ChildObserver.onStateChange(TcpServerBind.java:228) ~[reactor-netty-0.9.4.RELEASE.jar:0.9.4.RELEASE]
		at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:465) ~[reactor-netty-0.9.4.RELEASE.jar:0.9.4.RELEASE]
		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:90) ~[reactor-netty-0.9.4.RELEASE.jar:0.9.4.RELEASE]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:167) ~[reactor-netty-0.9.4.RELEASE.jar:0.9.4.RELEASE]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.45.Final.jar:4.1.45.Final]
		at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]

Sample If possible, please provide a test case or sample application that reproduces the problem. This makes it much easier for us to diagnose the problem and to verify that we have fixed it.

https://github.com/creatorKoo/error-report-actuator

start with this option -Dreactor.schedulers.defaultBoundedElasticSize=2 -Dreactor.schedulers.defaultBoundedElasticQueueSize=2

and

#!/bin/bash
# actuator request 100 times
curl http://127.0.0.1:11140/actuator/health &                                                                                                                                                                        curl http://127.0.0.1:11140/actuator/health &                                                                                                                                                                        
curl http://127.0.0.1:11140/actuator/health &       
curl http://127.0.0.1:11140/actuator/health &       
... (more than 100 lines) ...
curl http://127.0.0.1:11140/actuator/health

run this script and call /actuator/health and not error

and remove that info from bootstrap.yaml (set default)

spring:
  sleuth:
    reactor:
#      decorate-on-each: true
      decorate-on-each: false # this option fix error but lose useful functions.

run 100time’s script again and call /actuator/health and then error repeat.

I want to use decorate-on-each: true option with high load actuator call.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:18 (8 by maintainers)

github_iconTop GitHub Comments

1reaction
simonbaslecommented, Feb 12, 2020

In the meantime, as a workaround that should be removed as soon as the reactor-core fix is usable, you can set the task queue limit to ‘unbounded’ (Integer.MAX_VALUE):

-Dreactor.schedulers.defaultBoundedElasticQueueSize=2147483647
1reaction
simonbaslecommented, Feb 12, 2020

actually I don’t agree with your findings that it is ok without sleuth. Having sleuth added to the mix just seems to accelerate the problem, probably because a bunch of blocking health indicators get registered by spring-cloud, which will occupy more spots in the bounded queue…

In any case, there is an issue with boundedElasticScheduler(), obviously. We’re exploring an alternative implementation that should not have these recovery issues in reactor/reactor-core#2040

Read more comments on GitHub >

github_iconTop Results From Across the Web

Spring Boot Actuator Health Returning DOWN - Stack Overflow
In your Spring properties, set endpoints.health.sensitive = false . The /health endpoint will then return the list of various health indicators and you...
Read more >
Spring Cloud Sleuth
We maintain an example app where two Spring Boot services collaborate on an HTTP request. Sleuth configures these apps, so that timing of...
Read more >
Spring Boot Actuator | Baeldung
A quick intro to Spring Boot Actuators - using and extending the existing ones, configuration and rolling your own.
Read more >
Help, my Spring Boot info actuator endpoint is enabled, but I ...
So although you exposed the /info endpoint using the property management.endpoints.web.exposure.include you now have to enable the environment ...
Read more >
Exporting metrics to InfluxDB and Prometheus using Spring ...
This article shows how to use Spring Boot Actuator and Micrometer projects to ... management.metrics.web.server.auto-time-requests to true .
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Running tests with DirtiesContext fails with "(...) application context has been closed already"

Next page

Zipkin RabbitMQ Sender fails to reconnect with RabbitMQ broker