netty doesn't use maximum cpu even though it has room and things to do, much waiting at epollWaitNoTimerChange

I’m writing a network intensive WebFlux application. When receiving request, application requests and recieves something with another external server and then reply to original requester. I’m talking about the WebFlux application, and I’m using WebClient when call to external server.

Application’s performance is not that satisfiable. I think it should touch the maximum of CPU resource, maximum TPS at maximum CPU. But it shows low tps, cpu is just at 30 ~ 40%. Can we check why it does not use CPU any more to get more TPS, even though it has more room to execute more requests.

It waits a lot at many threads, like this… { “threadName”: “reactor-http-epoll-3”, “threadId”: 20, “blockedTime”: -1, “blockedCount”: 8, “waitedTime”: -1, “waitedCount”: 0, “lockName”: null, “lockOwnerId”: -1, “lockOwnerName”: null, “inNative”: true, “suspended”: false, “threadState”: “RUNNABLE”, “stackTrace”: [ { “methodName”: “epollWait”, “fileName”: “Native.java”, “lineNumber”: -2, “className”: “io.netty.channel.epoll.Native”, “nativeMethod”: true }, { “methodName”: “epollWait”, “fileName”: “Native.java”, “lineNumber”: 148, “className”: “io.netty.channel.epoll.Native”, “nativeMethod”: false }, { “methodName”: “epollWait”, “fileName”: “Native.java”, “lineNumber”: 141, “className”: “io.netty.channel.epoll.Native”, “nativeMethod”: false }, { “methodName”: “epollWaitNoTimerChange”, “fileName”: “EpollEventLoop.java”, “lineNumber”: 290, “className”: “io.netty.channel.epoll.EpollEventLoop”, “nativeMethod”: false }, { “methodName”: “run”, “fileName”: “EpollEventLoop.java”, “lineNumber”: 347, “className”: “io.netty.channel.epoll.EpollEventLoop”, “nativeMethod”: false }, {…

Expected behavior

CPU reaches 100% and shows good TPS

Actual behavior

CPU just stays at 30~40%, and shows low TPS

Steps to reproduce

reproducing codes : https://github.com/mouse500/perfwebf

/workloadwexcall : workload using external call

external call is implemented with simple nodejs server

jmeter to call above api with more than 200 threads.

Minimal yet complete reproducer code (or URL to code)

Netty version

netty-transport-native-epoll-4.1.52.Final

JVM version (e.g. java -version)

java 1.8.0_292 spring boot 2.3.4.RELEASE, WebFlux, WebClient

OS version (e.g. uname -a)

[mouse500]

Thank you for all

[mouse500]

@violetagg Ok, Thank you, I will try. Actaully, I already tried with various version of Spring Boot, and JDK version, Linux Distribion. Anway, will try with the latest version.

Now, I think this is about some other issue. I will look around something else more.

Thank you very much.