Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Resteasy-reactive runs out of direct buffer memory when processing large multipart messages

See original GitHub issue

Describe the bug

I have two Rest Reactive methods, one accepts File and another multipart of File and String. If running on resource limit machine(this behavior first arose on Github Actions), the second method fails with cryptic error while processing files of 2048 MB size, while the first handles larger(2054 MB) files without problems.

Expected behavior

Method should fail with more concise error or not fail at all.

Actual behavior

Method fails with this error:

2022-06-16T13:28:18.2721622Z 13:28:18,223 INFO  [app] 13:28:18,105 Request failed: java.io.IOException: java.io.IOException: java.lang.OutOfMemoryError: Direct buffer memory
2022-06-16T13:28:18.2726667Z 13:28:18,224 INFO  [app] 	at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput.readBlocking(VertxInputStream.java:253)
2022-06-16T13:28:18.2730751Z 13:28:18,224 INFO  [app] 	at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.readIntoBuffer(VertxInputStream.java:120)
2022-06-16T13:28:18.2735393Z 13:28:18,226 INFO  [app] 	at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.read(VertxInputStream.java:82)
2022-06-16T13:28:18.2736182Z 13:28:18,227 INFO  [app] 	at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.read(VertxInputStream.java:70)
2022-06-16T13:28:18.2795835Z 13:28:18,228 INFO  [app] 	at org.jboss.resteasy.reactive.server.core.multipart.MultiPartParserDefinition$MultiPartUploadHandler.parseBlocking(MultiPartParserDefinition.java:217)
2022-06-16T13:28:18.2806940Z 13:28:18,228 INFO  [app] 	at org.jboss.resteasy.reactive.server.handlers.FormBodyHandler.handle(FormBodyHandler.java:88)
2022-06-16T13:28:18.2811409Z 13:28:18,229 INFO  [app] 	at org.jboss.resteasy.reactive.server.handlers.FormBodyHandler.handle(FormBodyHandler.java:25)
2022-06-16T13:28:18.2816304Z 13:28:18,229 INFO  [app] 	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:141)
2022-06-16T13:28:18.2822693Z 13:28:18,229 INFO  [app] 	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$13.runWith(VertxCoreRecorder.java:545)
2022-06-16T13:28:18.2826225Z 13:28:18,230 INFO  [app] 	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
2022-06-16T13:28:18.2830935Z 13:28:18,230 INFO  [app] 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
2022-06-16T13:28:18.2836012Z 13:28:18,230 INFO  [app] 	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
2022-06-16T13:28:18.2888414Z 13:28:18,231 INFO  [app] 	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
2022-06-16T13:28:18.2892511Z 13:28:18,232 INFO  [app] 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2022-06-16T13:28:18.2898697Z 13:28:18,232 INFO  [app] 	at java.base/java.lang.Thread.run(Thread.java:829)
2022-06-16T13:28:18.2899383Z 13:28:18,232 INFO  [app] 	Suppressed: java.io.IOException: java.io.IOException: java.lang.OutOfMemoryError: Direct buffer memory
2022-06-16T13:28:18.2903591Z 13:28:18,232 INFO  [app] 		at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput.readBlocking(VertxInputStream.java:253)
2022-06-16T13:28:18.2904557Z 13:28:18,233 INFO  [app] 		at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.readIntoBuffer(VertxInputStream.java:120)
2022-06-16T13:28:18.2914225Z 13:28:18,233 INFO  [app] 		at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.close(VertxInputStream.java:148)
2022-06-16T13:28:18.2917218Z 13:28:18,233 INFO  [app] 		at org.jboss.resteasy.reactive.server.core.multipart.MultiPartParserDefinition$MultiPartUploadHandler.parseBlocking(MultiPartParserDefinition.java:214)
2022-06-16T13:28:18.2921708Z 13:28:18,233 INFO  [app] 		... 10 more
2022-06-16T13:28:18.2926611Z 13:28:18,233 INFO  [app] 	Caused by: java.io.IOException: java.lang.OutOfMemoryError: Direct buffer memory
2022-06-16T13:28:18.2931355Z 13:28:18,234 INFO  [app] 		at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput$2.handle(VertxInputStream.java:200)
2022-06-16T13:28:18.2934186Z 13:28:18,234 INFO  [app] 		at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput$2.handle(VertxInputStream.java:196)
2022-06-16T13:28:18.2938685Z 13:28:18,234 INFO  [app] 		at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
2022-06-16T13:28:18.2943012Z 13:28:18,235 INFO  [app] 		at io.vertx.core.http.impl.HttpEventHandler.handleException(HttpEventHandler.java:89)
2022-06-16T13:28:18.2947915Z 13:28:18,235 INFO  [app] 		at io.vertx.core.http.impl.Http1xServerRequest.handleException(Http1xServerRequest.java:635)
2022-06-16T13:28:18.2948945Z 13:28:18,235 INFO  [app] 		at io.vertx.core.http.impl.Http1xServerConnection.handleException(Http1xServerConnection.java:479)
2022-06-16T13:28:18.2953455Z 13:28:18,235 INFO  [app] 		at io.vertx.core.net.impl.VertxHandler.exceptionCaught(VertxHandler.java:136)
2022-06-16T13:28:18.2957910Z 13:28:18,235 INFO  [app] 		at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
2022-06-16T13:28:18.2964868Z 13:28:18,236 INFO  [app] 		at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
2022-06-16T13:28:18.2967697Z 13:28:18,236 INFO  [app] 		at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
2022-06-16T13:28:18.2972847Z 13:28:18,236 INFO  [app] 		at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
2022-06-16T13:28:18.2973933Z 13:28:18,238 INFO  [app] 		at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
2022-06-16T13:28:18.2980565Z 13:28:18,239 INFO  [app] 		at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
2022-06-16T13:28:18.2989152Z 13:28:18,239 INFO  [app] 		at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
2022-06-16T13:28:18.2994600Z 13:28:18,239 INFO  [app] 		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
2022-06-16T13:28:18.2998533Z 13:28:18,239 INFO  [app] 		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:177)
2022-06-16T13:28:18.3004558Z 13:28:18,239 INFO  [app] 		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
2022-06-16T13:28:18.3007277Z 13:28:18,239 INFO  [app] 		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
2022-06-16T13:28:18.3014467Z 13:28:18,240 INFO  [app] 		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
2022-06-16T13:28:18.3018855Z 13:28:18,240 INFO  [app] 		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
2022-06-16T13:28:18.3023471Z 13:28:18,240 INFO  [app] 		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
2022-06-16T13:28:18.3028808Z 13:28:18,240 INFO  [app] 		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2022-06-16T13:28:18.3032242Z 13:28:18,240 INFO  [app] 		... 2 more
2022-06-16T13:28:18.3035985Z 13:28:18,240 INFO  [app] 	Caused by: java.lang.OutOfMemoryError: Direct buffer memory
2022-06-16T13:28:18.3040268Z 13:28:18,240 INFO  [app] 		at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
2022-06-16T13:28:18.3044246Z 13:28:18,241 INFO  [app] 		at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
2022-06-16T13:28:18.3048818Z 13:28:18,241 INFO  [app] 		at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
2022-06-16T13:28:18.3049459Z 13:28:18,241 INFO  [app] 		at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:648)
2022-06-16T13:28:18.3053956Z 13:28:18,241 INFO  [app] 		at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:623)
2022-06-16T13:28:18.3054534Z 13:28:18,241 INFO  [app] 		at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:202)
2022-06-16T13:28:18.3058583Z 13:28:18,241 INFO  [app] 		at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:186)
2022-06-16T13:28:18.3059067Z 13:28:18,241 INFO  [app] 		at io.netty.buffer.PoolArena.allocate(PoolArena.java:136)
2022-06-16T13:28:18.3063255Z 13:28:18,242 INFO  [app] 		at io.netty.buffer.PoolArena.allocate(PoolArena.java:126)
2022-06-16T13:28:18.3072936Z 13:28:18,242 INFO  [app] 		at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:394)
2022-06-16T13:28:18.3077717Z 13:28:18,242 INFO  [app] 		at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
2022-06-16T13:28:18.3089443Z 13:28:18,246 INFO  [app] 		at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
2022-06-16T13:28:18.3099295Z 13:28:18,246 INFO  [app] 		at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140)
2022-06-16T13:28:18.3111835Z 13:28:18,251 INFO  [app] 		at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
2022-06-16T13:28:18.3123149Z 13:28:18,252 INFO  [app] 		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
2022-06-16T13:28:18.3128101Z 13:28:18,252 INFO  [app] 		... 8 more
2022-06-16T13:28:18.3139424Z 13:28:18,252 INFO  [app] 	[CIRCULAR REFERENCE:java.io.IOException: java.lang.OutOfMemoryError: Direct buffer memory]

How to Reproduce?

This steps should be run on memory limited machine(preferably, VM)

  1. git clone https://github.com/fedinskiy/quarkus-test-suite.git -b reproducer/multipart_oom tests
  2. cd tests
  3. mvn -Dall-modules -pl http/rest-client-reactive/ clean verify -Dit.test=LargeFileHandlingIT Test uploadMultipart fails, tests uploadBigFileThroughClient and uploadThroughClient work.

Output of uname -a or ver

4.18.0-372.9.1.el8.x86_64

Output of java -version

openjdk 11.0.13 2021-10-19 LTS(also temurin 11.0.15)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.7.6, 2.9.1 and ca836996b345a6cad93a79a6d8600c5dd809ed92

Build tool (ie. output of mvnw --version or gradlew --version)

3.8.3, 3.8.4 and 3.8.6

Additional information

CI logs: https://github.com/quarkus-qe/quarkus-test-suite/runs/6919232274?check_suite_focus=true

On 4G VM, which is described in the environment section, the test do not threw any errors, but hangs with warnings like these, but with different thread numbers:

12:55:30,733 INFO  [app] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "vert.x-eventloop-thread-2"
12:55:34,739 INFO  [app] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "vertx-blocked-thread-checker"
12:55:38,741 INFO  [app] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "executor-thread-0"

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:25 (24 by maintainers)

github_iconTop GitHub Comments

1reaction
geoandcommented, Sep 7, 2022

This is a pretty weird issue - I was able to track down the root cause (I’ll write it up later).

Now I need to see how it can be addressed

0reactions
rsvobodacommented, Sep 8, 2022
Read more comments on GitHub >

github_iconTop Results From Across the Web

Direct buffer memory - java - Stack Overflow
The actual memory buffers managed by DirectByteBuffer are not allocated in the heap. They are allocated using Unsafe.
Read more >
RESTEasy JAX-RS - JBoss.org
RESTEasy is installed and configured in different ways depending on which environment you are running in. If you are running in WildFly, RESTEasy...
Read more >
All configuration options - Quarkus
AWS Lambda Type Default AWS Lambda Common Type Default AWS Lambda Gateway REST API Type Default Agroal ‑ Database connection pool Type Default
Read more >
How to Fix java.lang.OufOfMemoryError: Direct Buffer Memory
3) One more thing, you can do to avoid java. lang. OutOfMemoryError: Direct buffer memory is increasing JVM default memory limit. By default,...
Read more >
2020 quarkus-fascicle-understanding-v1.pdf
on the APIs and technologies Quarkus exposes: CDI, RESTEasy, ... Reactive streams: Stream processing with non-blocking backpressure.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Exit code 137 even with `-Dquarkus.native.native-image-xmx=4G` when building native app

Next page

quarkus-cli failing to install