Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Memory Leak: Netty UsedMemory does not go down when sending large HTTP messages

See original GitHub issue

Expected behavior

No Memory leak for sending large HTTP messages

Actual behavior

Recently, we came to a situation where our app is going OutOfDirectMemory under load.

“io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 65536 byte(s) of direct memory (used: 3129351, max: 3145728) at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640)”

The situation seems to happen when E.g we assigned 2M of Direct Memory to Netty then we send 3 Http requests with 1MB body. As a result, Netty gets OutOfDirectMemory which is expected. But the problem we figured that after OutOfDirectMemory is thrown, the usedMemory value does not go down in PlatformDependent class in Netty which we think is a leak. So, if you continuously throw 3 requests, at some point your usedMemory will be equal to the max memory and you will never recover from the OutOfDirectMemory error.

Steps to reproduce

We tried to reproduce the situation with the basic Netty server example and we can reproduce it.

HttpSnoopServerHandler @Override public void channelRead(ChannelHandlerContext ctx, Object msg) { if ( !((FullHttpRequest)msg).decoderResult().isSuccess()) { return; }

 if(msg instanceof FullHttpRequest) {

    if (((FullHttpRequest) msg).decoderResult().isFailure()) {
        System.out.println("Got Failure in decode result");
    }
     try {

         FullHttpResponse response = new DefaultFullHttpResponse(HTTP_1_1, OK);

         ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE);
     } finally {
         ReferenceCountUtil.release(msg);
     }
 }

}

@Override public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) { cause.printStackTrace(); ctx.close(); }

HttpSnoonServerInitalizer public class HttpSnoopServerInitializer extends ChannelInitializer<SocketChannel> {

private final SslContext sslCtx;

public HttpSnoopServerInitializer(SslContext sslCtx) { this.sslCtx = sslCtx; }

@Override public void initChannel(SocketChannel ch) { ChannelPipeline p = ch.pipeline(); if (sslCtx != null) { p.addLast(sslCtx.newHandler(ch.alloc())); } p.addLast(new HttpRequestDecoder()); // Uncomment the following line if you don’t want to handle HttpChunks.
```
 p.addLast(new HttpResponseEncoder());
 p.addLast(new HttpObjectAggregator(3*1024*1024));
 // Remove the following line if you don't want automatic content compression.
 //p.addLast(new HttpContentCompressor());
 p.addLast(new HttpSnoopServerHandler());
```
} }
HttpSnoopServer // Configure the server. EventLoopGroup bossGroup = new NioEventLoopGroup(1); EventLoopGroup workerGroup = new NioEventLoopGroup(); try { ServerBootstrap b = new ServerBootstrap(); b.group(bossGroup, workerGroup) .channel(NioServerSocketChannel.class) .handler(new LoggingHandler(LogLevel.INFO)) .childHandler(new HttpSnoopServerInitializer(sslCtx));
```
     Channel ch = b.bind(PORT).sync().channel();

     System.err.println("Open your web browser and navigate to " +
             (SSL? "https" : "http") + "://127.0.0.1:" + PORT + '/');

     ch.closeFuture().sync();
 } finally {
     bossGroup.shutdownGracefully();
     workerGroup.shutdownGracefully();
 }
```
Client Side class to send the data public class HttpPOSTSender {

public static void main(String[] args) { int count = 10; ExecutorService executorService = Executors.newFixedThreadPool(count); for (int i = 1; i<=count; i ++) executorService.execute(()->{ try { URL request = new URL(new URL(“http://localhost:8082”), “http://localhost:8082/v1/topics/direct_memory_test/messages”); HttpURLConnection conn = (HttpURLConnection) request.openConnection(); conn.setDoInput(true); conn.setUseCaches(false); conn.setConnectTimeout(5000); conn.setReadTimeout(32000); conn.setInstanceFollowRedirects(false); conn.setRequestMethod(“POST”); conn.setDoOutput(true); conn.setRequestProperty(“Content-type”,“x-protobuf”); conn.setRequestProperty(“Authorization”, "Bearer " + “kjfajdalsjfladjlkajlajsklajoi3ji3jrl2kn3rnfkwdnlfnalnalknflakfjalskjfalsjalsjaflkjfo3i4jorjrj3lk4tnrklendksdavnadkjldnvkadnvkasndfkaldnvkadnva;lkvalk;kjaldvklafjlk43klnrk4n34knk3nknk5j34iediid9sivpsafjalkvdanlkjfalskdjfaklwdjflaks;dkjaklfjdakljvdsalkdfnadkjjfkldsjafaeljfdkjfklanjklsfdjs”); //conn.setRequestProperty(“Authorization”, "Bearer " + “eyJ4NXUiOiJ”); try (OutputStream postStream = conn.getOutputStream()) { postStream.write(new byte[1000100]); } System.out.println("Result: " + conn.getResponseCode()); } catch (Exception ex) { ex.printStackTrace(); }

});
```
 try {
     Thread.sleep(2000);
 } catch (InterruptedException e) {
     e.printStackTrace();
 }
 executorService.shutdown();
```
} }

Steps:

Run the HttpSnoopServer
Run the HttpPOSTSender class once
Give a breakpoint inside incrementMemoryCounter of PlatformDependent class of Netty
Run the HttpPOSTSender class again
Check the “usedMemory” value which should go up from where it started

The configuration that we used to run the HttpSnoopServer is -Xms10M -Xmx10M -XX:MaxDirectMemorySize=3M -Dio.netty.leakDetectionLevel=PARANOID -Dio.netty.leakDetection.maxRecords=32