EventStream fails with Spring Boot 2.3.5

Very similar to this issue: https://github.com/codecentric/spring-boot-admin/issues/672

My spring-boot-admin server application was created using ‘start.spring.io’ and the instructions from the docs: https://codecentric.github.io/spring-boot-admin/current/

My setup has ‘no security’. Here’s a link to my test application:

https://github.com/kdvolder/spring-boot-docker-fortune-teller/tree/cd7741d7c61de9142c58321da5bc4af61e0327e0

Description of the problem:

• server seems to run mostly fine. But on occasion we get the ‘Server Conection Failed’ warning banner. If we keep reloading the page, it seems to eventually clear itself.

Things also seem to be working mostly fine as long as the services shown in the main page remain stable. But if we start or stop services then the error often reappears.

So a method that I have to more or less reliably reproduce this is as follows:

• start all the apps in my test application. (I run them from STS boot dash, but you can also run them on the CLI, I suggeest running each app in its own terminal so you can control easily when to start/stop them individually. Command to start them is `./mvnw -Dmaven.test.skip=true -Dspring.profiles.active=local spring-boot:run
• There are 4 apps in total fortune-eureka, fortune-admin-server, fortune-ui, fortune-service.

When all apps are running access admin-server on port 8000 (http://localhost:8000).

Initially the error may appear, refreshing the page a few times will ‘fix it’.

Now stop the ‘fortune-service’.

Very often I immediately see the error reapear in boot-admin ui (which is still open in browser).

If it does not appear by itself, then usually appears when I refresh the page.

Again refreshing the page a few times eventually fixes it.

Start ‘fortune-service’ again and the error comes back.

This process can be repeated and will reproduce the error almost every time.

I’ve also seen this error stacktrace in the of spring-boot-admin server. This error doesn’t allways get logged but it seems connected.

Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
	at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) ~[netty-transport-native-unix-common-4.1.53.Final.jar:4.1.53.Final]
	at io.netty.channel.unix.Socket.finishConnect(Socket.java:251) ~[netty-transport-native-unix-common-4.1.53.Final.jar:4.1.53.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:673) ~[netty-transport-native-epoll-4.1.53.Final-linux-x86_64.jar:4.1.53.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650) ~[netty-transport-native-epoll-4.1.53.Final-linux-x86_64.jar:4.1.53.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:530) ~[netty-transport-native-epoll-4.1.53.Final-linux-x86_64.jar:4.1.53.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465) ~[netty-transport-native-epoll-4.1.53.Final-linux-x86_64.jar:4.1.53.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.53.Final-linux-x86_64.jar:4.1.53.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.53.Final.jar:4.1.53.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.53.Final.jar:4.1.53.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.53.Final.jar:4.1.53.Final]
	at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]

Note: Perhaps this is a ‘Application Error/problem’, as sugested in the other bug as well. This part of the trace:

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) 

Makes me think maybe it has something to do with more than one thing trying to use the same ‘singleTread’ at the same time and so one of them is ‘rejected’.

If it is indeed some kind of configuration problem… perhaps it would be good to document the need for configuring this (whatever ‘this’ is) in docs somewhere.

[kdvolder]

Okay so I change spring-boot version from 2.3.4 to 2.3.5 in the parent pom of spring-boot-admin and now the eureka sample has same problem as my own. So it looks like there’s some change in spring boot or its dependencies that’s causing this.

[kdvolder]

I tried to use the sample from here: https://github.com/codecentric/spring-boot-admin/tree/master/spring-boot-admin-samples/spring-boot-admin-sample-eureka

That doesn’t have the problem. So wanted to try and narrow down what is the difference.

To make both of them as much similar as possible I made few changes on both side:

• spring-boot-admin-sample-eureka

  • profile: insecure
  • commented out spring-security dependency (as I don’t have that).
  • commented out security related beans that don’t compile anymore

• my own sample:

  • copied the application.yml exactly from admin-eureka-sample
  • made the annotation on main class exactly same as on admin-eureka-sample

This made no difference.

Finally changed version of boot from 2.3.5 (what I had) to 2.3.4 (what boot-admin-sample-eureka has)… and seems like the problem went away.