HTTP2 performance on client side

Background:

We are trying to adopt Netty based http2(SSL) to our project. The use case is: a router connects many data nodes and send/receive requests. The router is a Netty http2 based client and data nodes are netty http2 based servers.

The initial implementation is done.

We did a performance test which sent GET requests to all data nodes from the router. The GET request size is 100Bytes and response size is 1MB. QPS is 100. I looked at our internal metrics and noticed that data nodes performance are good, but router performance is terrible(P95 latency > 1k ms, same datacenter). The huge latency mainly comes from stream send-to-response time(start from frame’s writeAndFlush() to response arrived in our stream channel handler). CPU and memory usage are low: around 10%. Network bandwidth is good. Buffer size: 1MB.

We tried to use EpollEventLoopGroup and tune number of connections to each data node, but get almost same result.

Does anyone has similar http2 client performance experience? Is there any netty metric I can look at? I am particularly interested in the stream round trip time(from request leaves netty to response comes back).

Any help will be appreciated.

Expected behavior

Latency under 200ms.

Actual behavior

Latency > 1K ms.

Steps to reproduce

not easy.

Minimal yet complete reproducer code (or URL to code)

Netty version

4.1.42.Final

JVM version (e.g. java -version)

java version “1.8.0_121”

OS version (e.g. uname -a)

Linux 3.10.0-957.12.2.el7.x86_64 #1 SMP Fri Apr 19 21:09:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[zzmao]

@ninja- Get some numbers just now!

I changed maxFrameSize, window size, Netty recv buffer and OS TCP max_rmem to 1MB. These change reduced GET P95 latency to 30ms from 1000ms+!

With these changes, CPU usage go up to 35% from 5%.

[ninja-]

YourKit profiler maybe if that’s what you are asking