Large number of PoolThreadCache garbage objects

We are using grpc-java version 1.16.0 in Apache Ratis for RPC communication. Apache Ratis is a java implementation of Raft consensus protocol and uses grpc for server-server and client-server communication. In the scenario there are ~10KB requests coming from the client to the server and the server is replicating those requests to other servers. There are thousands of these requests coming gradually to the server from the client. There are other small sized metadata related communications as well.

We are currently seeing a lot of GCs getting triggered and lot of garbage PoolThreadCache objects. Here is a snippet of the major unreachable heap objects.

org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache self 126Kb (< 0.1%), 2,021 object(s)
	.tinySubPageDirectCaches ↘ 173,584Kb (24.5%), 2,021 reference(s)
		org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$MemoryRegionCache[] self 284Kb (< 0.1%), 2,021 object(s)
			{array elements} ↘ 173,300Kb (24.4%), 64,672 reference(s)
				org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache self 2,021Kb (0.3%), 64,672 object(s)
					.queue ↘ 171,279Kb (24.1%), 64,672 reference(s)
						org.apache.ratis.thirdparty.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue self 40,925Kb (5.8%), 64,672 object(s)
							.buffer ↘ 130,354Kb (18.4%), 64,672 reference(s)
								Object[] self 130,354Kb (18.4%), 64,672 object(s)

	.tinySubPageHeapCaches ↘ 173,584Kb (24.5%), 2,021 reference(s)
		org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$MemoryRegionCache[] self 284Kb (< 0.1%), 2,021 object(s)
			{array elements} ↘ 173,300Kb (24.4%), 64,672 reference(s)
				org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache self 2,021Kb (0.3%), 64,672 object(s)
					.queue ↘ 171,279Kb (24.1%), 64,672 reference(s)
						org.apache.ratis.thirdparty.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue self 40,925Kb (5.8%), 64,672 object(s)
							.buffer ↘ 130,354Kb (18.4%), 64,672 reference(s)
								Object[] self 130,354Kb (18.4%), 64,672 object(s)

	.smallSubPageDirectCaches ↘ 13,641Kb (1.9%), 2,021 reference(s)
		org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$MemoryRegionCache[] self 63Kb (< 0.1%), 2,021 object(s)
			{array elements} ↘ 13,578Kb (1.9%), 8,084 reference(s)
				org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache self 252Kb (< 0.1%), 8,084 object(s)
					.queue ↘ 13,325Kb (1.9%), 8,084 reference(s)
						org.apache.ratis.thirdparty.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue self 5,115Kb (0.7%), 8,084 object(s)
							.buffer ↘ 8,210Kb (1.2%), 8,084 reference(s)
								Object[] self 8,210Kb (1.2%), 8,084 object(s)

	.smallSubPageHeapCaches ↘ 13,641Kb (1.9%), 2,021 reference(s)
		org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$MemoryRegionCache[] self 63Kb (< 0.1%), 2,021 object(s)
			{array elements} ↘ 13,578Kb (1.9%), 8,084 reference(s)
				org.apache.ratis.thirdparty.io.netty.buffer.PoolThreadCache$SubPageMemoryRegionCache self 252Kb (< 0.1%), 8,084 object(s)
					.queue ↘ 13,325Kb (1.9%), 8,084 reference(s)
						org.apache.ratis.thirdparty.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue self 5,115Kb (0.7%), 8,084 object(s)
							.buffer ↘ 8,210Kb (1.2%), 8,084 reference(s)
								Object[] self 8,210Kb (1.2%), 8,084 object(s)

	.normalDirectCaches ↘ 5,699Kb (0.8%), 2,021 reference(s)
	
	.normalHeapCaches ↘ 5,699Kb (0.8%), 2,021 reference(s)

	.freed ↘ 31Kb (< 0.1%), 2,021 reference(s)

It seems like there is lot of garbage being generated via thread caches.

[zandegran]

We are also affected by this memory leak large amounts of io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue.buffer objects. Upto 30% of the heap

[MajorHe1]

@ejona86 Thanks for your reply!! The screen shot indeed doesn’t look problematic, as you said. The problem is with the amount of this kind of objects that we see in that screen shot. The following screen shot shows it better (different dump): I think that many of these Object[] instances are just arrays of null values, and they are also related to these caches. I’ve just now realised that the service didn’t run enough time to really demonstrate the memory leak. I’ll monitor if for several more days.

Hello, is there any process to solve this problem?