How to tune Netty to reduce writeAndFlush time

Expected behavior

We are developing a Netty based HTTP2 server to replace exiting Java socket based server. We expect Netty + HTTP2 can give us better performance/latency.

Actual behavior

However, in a recent performance test, where multiple clients send large amount of GET requests to server, Netty+HTTP/2 behaved worse than socket implementation.

We tried one request per http/2 connection and http2 multiplexing. Both were worse than socket implementation. We also tried 1MB for GET and 4MB for GET, both were worse than socket implementation.

During the performance test, data is in memory and none of CPU/NIC saturated. (NIC/CPU usage lower than socket implementation. )

Our metrics showed that the majority of the time is in writeAndFlush: from writeAndFlush to listener invoked.

Is there any parameter we can tune to get better latency? (We have increased TCP buffer to 4MB)

Steps to reproduce

Minimal yet complete reproducer code (or URL to code)

Netty version

JVM version (e.g. java -version)

java -version java version “1.8.0_121” Java™ SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot™ 64-Bit Server VM (build 25.121-b13, mixed mode)

OS version (e.g. uname -a)

Linux

[xiaoxipanda]

You can reference cassandra optimization FlushConsolidationHandler implement.

Message#Dispatcher

[normanmaurer]

Its really impossible to give any guidance without more details and profiling data. All we can do at the moment is basically guess