OOM when uploading large files
See original GitHub issueI was recently guarding our app with security checks and decided to limit the size of the http request. After this I found that server dies with OOM when I try to upload few large files.
In my test I:
- Run the server;
- Upload 3 times the same file with 1.4GB size;
- Server throws exceptions below;
java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:768)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:744)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:227)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:147)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:327)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2120)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1325)
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Profiles result after 3 uploads:
34008984 29.13% 65 io.netty.buffer.PooledSlicedByteBuf
27497824 23.55% 50 io.netty.util.Recycler$DefaultHandle
17348936 14.86% 36 io.netty.buffer.PooledUnsafeDirectByteBuf
12710968 10.89% 27 java.nio.DirectByteBuffer
10978400 9.40% 26 io.netty.handler.codec.http.DefaultHttpContent
5043456 4.32% 16 javax.net.ssl.SSLEngineResult
3215928 2.75% 5 byte[]
1961568 1.68% 4 io.netty.buffer.PoolSubpage[]
1150160 0.99% 2 java.lang.String
702952 0.60% 1 java.util.ArrayDeque
Server eats 8 GB after 3 tries.
The main difference I found between OOM server and normal server (server that frees the memory between large file uploads) is HttpObjectAggregator
.
Server that has OOM has:
.addLast("HttpsObjectAggregator",
new HttpObjectAggregator(holder.limits.webRequestMaxSize, true))
web.request.max.size=5242880
Server that frees the memory between file uploads:
.addLast("HttpsObjectAggregator",
new HttpObjectAggregator(Integer.MAX_VALUE, true))
Profiles result after 3 uploads without request size limit:
bytes percent samples top
---------- ------- ------- ---
19270216 16.74% 95 byte[]
17301648 15.03% 9 byte[] (out)
15983536 13.88% 56 io.netty.buffer.CompositeByteBuf$Component
15472152 13.44% 53 io.netty.handler.codec.http.DefaultHttpContent
12619656 10.96% 46 long[]
8130168 7.06% 32 java.lang.Object[]
7883232 6.85% 29 javax.net.ssl.SSLEngineResult
3072624 2.67% 8 java.lang.String
2181960 1.90% 7 int[]
1889664 1.64% 5 short[]
1754400 1.52% 8 java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
1731048 1.50% 8 io.netty.buffer.PooledSlicedByteBuf
1620920 1.41% 7 java.lang.String[]
1332704 1.16% 5 io.netty.util.Recycler$DefaultHandle
978648 0.85% 5 io.netty.util.ResourceLeakDetector$Record
Similar issue - https://github.com/netty/netty/issues/3559.
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment (build 11.0.1+13-Ubuntu-3ubuntu116.04ppa1)
OpenJDK 64-Bit Server VM (build 11.0.1+13-Ubuntu-3ubuntu116.04ppa1, mixed mode, sharing)
<netty.version>4.1.34.Final</netty.version>
<netty.boring.ssl.version>2.0.22.Final</netty.boring.ssl.version>
Pipeline is pretty complex, however this is main handlers:
.addLast(holder.sslContextHolder.sslCtx.newHandler(ch.alloc()))
.addLast("HttpsServerCodec", new HttpServerCodec())
.addLast("HttpsObjectAggregator", new HttpObjectAggregator(holder.limits.webRequestMaxSize, true))
.addLast("HttpsChunkedWrite", new ChunkedWriteHandler())
Issue Analytics
- State:
- Created 4 years ago
- Comments:24 (24 by maintainers)
Top Results From Across the Web
OOM while uploading large file - java - Stack Overflow
I need to upload a very large file from my machine to a server. (a few GB) Currently, I tried the below approach...
Read more >Out of Memory Error When Uploading Large Files Through ...
I have some javascript that sends two videos to the handler.ashx page, then performs the upload. It works fine on smaller files, but...
Read more >Java Large Files - Efficient Processing - amitph
Guide to Optimal ways of Java Large Files Processing. Compare between the fasted and the most memory efficient ways to read and write...
Read more >File Uploads and Downloads with Spring Boot - Niels.nu
In this blog post I'm going to explain how this can be implemented in Spring in a way that can scale to very...
Read more >Upload Large Files in C# | Scattered Code
Uploading files these days is pretty straightforward in just about any web framework and programming language, however, when files get big or many...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@doom369 @rkapsi from a quick look I think this line must be removed:
https://github.com/netty/netty/pull/8793/files#diff-0522c090d2ed17d5b357324a06ace7a9R403
@doom369 what happens if you revert https://github.com/netty/netty/pull/8793 ? Do you get the 413 error ?