Connection prematurely closed BEFORE response

Hi,

I’m using SBA 2.4.1 in combination with spring boot 2.4.5 in a Docker/Tomcat environment. SBA runs in a plain openjdk container and my actual spring boot applications (2) run inside a tomcat docker container.

As I run both of them, I got in an irregular time interval status change mails (link this example below) caused by this exception below. There as I’m the opinion, that the actual applications works fin all the time. The appended exception is from the SBA container.

Mail Example:

app-api (ff3c712f6338) is OFFLINE

Instance ff3c712f6338 changed status from UP to OFFLINE

Status Details

exception
org.springframework.web.reactive.function.client.WebClientRequestException
message
Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
Registration

Service Url	http://app:8080/api
Health Url	http://app:8080/api/actuator/health
Management Url	http://app:8080/api/actuator

My configuration looks like this:

SBA:

spring.security.user.name=admin
spring.security.user.password=<password>
server.port=8000

Spring Boot Application:

spring.boot.admin.client.url=http://admin-server:8000
spring.boot.admin.client.instance.service-base-url=http://app:8080
spring.boot.admin.client.username=admin
spring.boot.admin.client.password=admin
spring.boot.admin.client.instance.metadata.user.name=client
spring.boot.admin.client.instance.metadata.user.password=<password>

Do I have anything misconfigured or is this a bug?

The actual Exception inside SBA:

2021-04-24 11:23:44.946  WARN 1 --- [or-http-epoll-2] r.netty.http.client.HttpClientConnect    : [id:f901f883-15, L:/172.28.0.2:40130 ! R:app/172.28.0.3:8080] The connection observed an error

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response

2021-04-24 11:23:44.973  INFO 1 --- [or-http-epoll-2] d.c.b.a.server.services.StatusUpdater    : Couldn't retrieve status for Instance(id=ff3c712f6338, version=7, registration=Registration(name=app-api, managementUrl=http://app:8080/api/actuator, healthUrl=http://app:8080/api/actuator/health, serviceUrl=http://app:8080/api, source=http-api), registered=true, statusInfo=StatusInfo(status=UP, details={mongo={status=UP, details={version=4.4.5}}, db={status=UP, details={database=MySQL, validationQuery=isValid()}}, diskSpace={status=UP, details={total=420696522752, free=318616391680, threshold=10485760, exists=true}}, ping={status=UP}, redis={status=UP, details={version=5.0.12}}}), statusTimestamp=2021-04-24T09:20:31.726866Z, info=Info(values={git={branch=origin/master, commit={id=725df9c, time=2021-04-24T09:08:38Z}}}), endpoints=Endpoints(endpoints={mappings=Endpoint(id=mappings, url=http://app:8080/api/actuator/mappings), httptrace=Endpoint(id=httptrace, url=http://app:8080/api/actuator/httptrace), caches=Endpoint(id=caches, url=http://app:8080/api/actuator/caches), loggers=Endpoint(id=loggers, url=http://app:8080/api/actuator/loggers), logfile=Endpoint(id=logfile, url=http://app:8080/api/actuator/logfile), configprops=Endpoint(id=configprops, url=http://app:8080/api/actuator/configprops), scheduling=Endpoint(id=scheduling, url=http://app:8080/api/actuator/scheduling), health=Endpoint(id=health, url=http://app:8080/api/actuator/health), env=Endpoint(id=env, url=http://app:8080/api/actuator/env), info=Endpoint(id=info, url=http://app:8080/api/actuator/info)}), buildVersion=null, tags=Tags(values={}))

org.springframework.web.reactive.function.client.WebClientRequestException: Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
	at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) ~[spring-webflux-5.3.6.jar!/:5.3.6]
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ Request to GET health [DefaultWebClient]
Stack trace:
		at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141) ~[spring-webflux-5.3.6.jar!/:5.3.6]
		at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:70) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4150) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:221) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:221) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:221) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:224) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:273) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:413) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:250) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:491) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:299) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:97) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:189) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[reactor-core-3.4.5.jar!/:3.4.5]
		at reactor.netty.http.client.HttpClientConnect$HttpObserver.onUncaughtException(HttpClientConnect.java:358) ~[reactor-netty-http-1.0.6.jar!/:1.0.6]
		at reactor.netty.ReactorNetty$CompositeConnectionObserver.onUncaughtException(ReactorNetty.java:647) ~[reactor-netty-core-1.0.6.jar!/:1.0.6]
		at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onUncaughtException(DefaultPooledConnectionProvider.java:214) ~[reactor-netty-core-1.0.6.jar!/:1.0.6]
		at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnection.onUncaughtException(DefaultPooledConnectionProvider.java:462) ~[reactor-netty-core-1.0.6.jar!/:1.0.6]
		at reactor.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:290) ~[reactor-netty-http-1.0.6.jar!/:1.0.6]
		at reactor.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:74) ~[reactor-netty-core-1.0.6.jar!/:1.0.6]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.handler.codec.http.HttpContentDecoder.channelInactive(HttpContentDecoder.java:235) ~[netty-codec-http-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) ~[netty-codec-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[netty-codec-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:311) ~[netty-codec-http-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:818) ~[netty-transport-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[netty-common-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar!/:4.1.51.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.51.Final.jar!/:4.1.51.Final]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.51.Final.jar!/:4.1.51.Final]
		at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response

[erikpetzold]

Hi everyone,

could you please try if enabling retry solves the problem? See https://codecentric.github.io/spring-boot-admin/current/#_configuration_options

There are the options spring.boot.admin.monitor.default-retries and spring.boot.admin.monitor.retries.* to configure the number of retries (default is 0, here we need retry for health endpoint).

So please add spring.boot.admin.monitor.default-retries=5 or spring.boot.admin.monitor.retries.health=5 to your config.

Does not remove the connection issue, but hopefully fixes the status changes.

[MrZhengliang]

Current project: Spring Boot version: 2.4.2 Reactor netty version: 1.0.3 spring-boot-admin version: 2.4.3 When we check health, the following exception logs often appear in spring boot admin: reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response

spring-boot-admin application.yml spring: boot: admin: monitor: status-interval: 30000 status-lifetime: 30000