Hints on setup (gc and threadpool)

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.6.1

What did you expect to see?

I’m starting to build another grpc service that’s supposed to handle something like 5k message per second (on commodity hardware) on average.

Messages weigh about 1k.

What’s a good setup when it comes to:

• gc (we give it up to 8GB using are configuring -XX:+UseG1GC, -XX:ParallelGCThreads=8, -XX:ConcGCThreads=2)
• thread-pool (we’re now using new NioEventLoopGroup(16)

Is there any recommended setup?

Thanks

[carl-mastrangelo]

I recommend using fewer threads in your NioEventLoop, and more in your ManagedChannel / Server executor. gRPC is usually less of your work load (i.e. your business logic) so it doesn’t need so many resources. I also recommend setting -Xmx and -Xms to the same value, and as high as you can allow. This makes memory management (from a sysadmin POV) easier. We typically set it to around 8GB in our benchmarks.

A lot of the times getting the system to be fast involves running it and finding the slow parts. Once you know the slow parts, we (gRPC team) can give more specific advice about what to optimize. You can use a Java profiler (like FlightRecorder, YourKit, JProfiler, etc. to get this information, so that you can see where to optimize next.

[carl-mastrangelo]

If the application logic you are doing is non blocking, you can try using the directExecutor() on your ServerBuilder which will avoid the wakeup. Otherwise, this behavior is expected. Youll need to do much more work in order to get the batching benefits. Right now, if the work level is too low, the server will be constantly waking up the Nio thread.