Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Memory leak in the webserver

See original GitHub issue

Environment Details

Helidon Version: 2.2.1
Helidon SE or Helidon MP: Helidon SE
JDK version: 15
OS: mac 11.3.1
Docker version (if applicable): 20.10.6, build 370c289

Problem Description

I use the following libraries import io.helidon.webserver.*; import io.helidon.common.http.DataChunk; import io.helidon.common.reactive.IoMulti; import io.helidon.common.reactive.Multi; to send a large file over http, and got a java.lang.OutOfMemoryError: Java heap space error:

    | java.lang.OutOfMemoryError: Java heap space
    |        at java.base/java.io.InputStream.readNBytes(Unknown Source)
    |        at io.helidon.common.reactive.MultiFromInputStream$InputStreamSubscription.submit(MultiFromInputStream.java:97)
    |        at io.helidon.common.reactive.MultiFromInputStream$InputStreamSubscription.trySubmit(MultiFromInputStream.java:146)
    |        at io.helidon.common.reactive.MultiFromInputStream$InputStreamSubscription.request(MultiFromInputStream.java:142)
    |        at io.helidon.common.reactive.MultiMapperPublisher$MapperSubscriber.request(MultiMapperPublisher.java:104)
    |        at io.helidon.common.reactive.MultiTappedPublisher$MultiTappedSubscriber.request(MultiTappedPublisher.java:276)
    |        at io.helidon.webserver.BareResponseImpl.onSubscribe(BareResponseImpl.java:328)
    |        at io.helidon.media.common.MessageBodyContext$EventingSubscriber.onSubscribe(MessageBodyContext.java:324)
    |        at io.helidon.webserver.accesslog.SizeLogEntry$ByteCountingSubscriber.onSubscribe(SizeLogEntry.java:112)
    |        at io.helidon.common.reactive.MultiTappedPublisher$MultiTappedSubscriber.onSubscribe(MultiTappedPublisher.java:214)
    |        at io.helidon.common.reactive.MultiMapperPublisher$MapperSubscriber.onSubscribe(MultiMapperPublisher.java:62)
    |        at io.helidon.common.reactive.MultiFromInputStream.subscribe(MultiFromInputStream.java:51)
    |        at io.helidon.common.reactive.MultiMapperPublisher.subscribe(MultiMapperPublisher.java:42)
    |        at io.helidon.common.reactive.MultiTappedPublisher.subscribe(MultiTappedPublisher.java:86)
    |        at io.helidon.webserver.accesslog.SizeLogEntry$ByteCountingPublisher.subscribe(SizeLogEntry.java:96)
    |        at io.helidon.media.common.MessageBodyContext$EventingPublisher.subscribe(MessageBodyContext.java:293)
    |        at io.helidon.webserver.Response.lambda$send$2(Response.java:206)
    |        at io.helidon.webserver.Response$$Lambda$722/0x0000000100373ab8.run(Unknown Source)
    |        at io.helidon.webserver.Response$SendLockSupport.execute(Response.java:375)
    |        at io.helidon.webserver.Response.send(Response.java:204)
    |        at io.helidon.webserver.Response.send(Response.java:192)

Steps to reproduce

code snippet for sending a large file over http: BufferedInputStream catStdOut = new BufferedInputStream(process.getInputStream());

Multi<DataChunk> ioMulti = IoMulti.multiFromStreamBuilder(catStdOut).byteBufferSize(8 * 1024 * 1024).build()
                                .map(byteBuffer -> DataChunk.create(true, byteBuffer))

                                .onTerminate(() -> {
                                    CompletableFuture.runAsync(() -> {
                                       
                                       ...          
                                    });
                                });

                        res.status(200);

                        res.send(ioMulti)
                                .onTerminate(() -> {
                                    try {
                                        catStdOut.close();
                                    } catch (IOException e) {
                   ...
                                    }
                                });

Issue Analytics

State:
Created 2 years ago
Comments:7 (5 by maintainers)

Top GitHub Comments

1reaction

danielkeccommented, Jun 9, 2021

We will need to introduce back-pressure to BareResponseImpl in order to let Netty flush when using single threaded publishers:

diff --git a/webserver/webserver/src/main/java/io/helidon/webserver/BareResponseImpl.java b/webserver/webserver/src/main/java/io/helidon/webserver/BareResponseImpl.java
--- a/webserver/webserver/src/main/java/io/helidon/webserver/BareResponseImpl.java	
+++ b/webserver/webserver/src/main/java/io/helidon/webserver/BareResponseImpl.java	
@@ -330,7 +330,7 @@
     @Override
     public void onSubscribe(Flow.Subscription subscription) {
         this.subscription = subscription;
-        subscription.request(Long.MAX_VALUE);
+        subscription.request(2);
     }
     @Override
@@ -440,6 +440,7 @@
                         }
                     });
                     data.release();
+                    subscription.request(1);
                     LOGGER.finest(() -> log("Data chunk sent with result: %s", future.isSuccess()));
                 })
                 .addListener(completeOnFailureListener("Failure when sending a content!"))

0reactions

danielkeccommented, Jun 10, 2021

Another workaround is limit buffer size, with size under 1 MB Netty flushing works fine

Top Results From Across the Web

How to Find Memory Leaks in Websites and Web Applications

In computer science, a memory leak is a leak of resources when computer software incorrectly manages memory allocation. A memory leak occurs ...

Fixing memory leaks in web applications | Read the Tea Leaves

One of these problems is memory leaks. A poorly-coded SPA can easily eat up megabytes or even gigabytes of memory, continuing to gobble...

Why do some webservers complain about memory leaks they ...

Memory leaks are detected by scanning the web server's worker thread pool for ThreadLocal variables. If a ThreadLocal variable was found the variable's ......

Memory leak detection - How to find, eliminate, and avoid

Another important way to prevent memory leaks is to write code which disposes of unneeded resources. Nearly all languages include resource types ...

Troubleshooting native memory leak in an IIS 7.x Application ...

When the memory leak in an IIS application pool occurs, increasing physical memory (RAM) does not effective because the memory in this scenario ......