HelidonSE - DataChunkInputStream never reads bytes

When attempting to receive whole request body as InputStream and reading it afterwards, underlying DataChunkInputStream never emits bytes. Nothing happens after get() call inside read method (https://github.com/oracle/helidon/blob/bef4b1e3f7/media/common/src/main/java/io/helidon/media/common/DataChunkInputStream.java#L131).

P.S. Also looking at tests and JerseySupport it’s not really clear how to glue together async API with blocking API.

Environment Details

• Helidon Version: 2.4.1
• Helidon SE
• JDK version: openjdk version “11.0.13” 2021-10-19
• OS: Arch Linux x86_64 Kernel: 5.16.3-arch1-1
• Docker version (if applicable): N/A

---

Problem Description

For some reason it’s not possible to convert DataChunks from req.content() as InputStream (InputStreamBodyReader is registered). Similar pattern that can be used when converting to String doesn’t apply to InputStream. Nothing is printed to debug log. Calling await(Duration) doesn’t help either and no InterruptedException is produced

Steps to reproduce

What works: Get a string from request and print it:

request.content().as(String.class)
                .thenAccept(s -> LOG.info("Got string {}", s));

When transforming to InputStream publisher never yields chunks and as a result no LOG message:

request.content().as(InputStream.class)
                .thenAccept(is -> {
                    try {
                        LOG.info("Got string {}", new String(is.readAllBytes()));
                    } catch (IOException e) {
                        LOG.error("Cannot transfer bytes", e);
                    }
                });

P.S. When trying .get, .await and other methods which block, underlying future hangs indefinitely (regardless if as(String.class) or as(InputStream.class) was called previously) instead of producing result.

[danielkec]

We have a lot in common with RxJava as most of our rx operators were contributed by David Karnok 🙂. I don’t have much experience with Okio but I think it helps a lot to avoid InputStream in this use-case:

        WebServer.builder(Routing.builder()
                        .put("/proxy", (req, res) -> {
                            WebClient.builder()
                                    .baseUri("http://localhost:8080")
                                    .build()
                                    .put()
                                    .path("/receiver")
                                    .submit(req.content())// Just connecting the reactive streams SERVER >> CLIENT
                                    .flatMapSingle(webClientResponse -> webClientResponse.content().as(String.class))
                                    .forSingle(receiverResponse ->
                                            res.send("PROXY: Receiver received the data: " + receiverResponse))
                                    .exceptionally(res::send);
                        })
                        .put("/receiver", (req, res) -> {
                            req.content()
                                    .collect(StringBuilder::new, (sb, dataChunk) -> {
                                        sb.append(new String(dataChunk.bytes()));
                                        dataChunk.release();
                                    })
                                    .map(StringBuilder::toString)
                                    .forSingle(msg -> {
                                        LOG.log(Level.INFO, "RECEIVER: I've got the data: " + msg);
                                        res.send("RECEIVER: I've got the data: " + msg);
                                    })
                                    .exceptionally(res::send);
                        })
                        .build())
                .config(Config.create().get("server"))
                .build()
                .start()
                .peek(ws -> {
                    System.out.println("WEB server is up! curl -X PUT -d 'hello world' http://localhost:" + ws.port() + "/proxy");
                    ws.whenShutdown().forSingle(w -> System.out.println("WEB server is DOWN. Good bye!"));
                })
                .onError(t -> {
                    System.err.println("Startup failed: " + t.getMessage());
                    t.printStackTrace(System.err);
                })
                .ignoreElement();

This is very simple example, it can get much more interesting with multipart, we have customers who are streaming gigabytes of data with Helidon SE like that.

[danielkec]

Its reactive api, you are blocking the thread which should be feeding you the data later over the InputStream by readAllBytes() getting yourself deadlocked. InputStream isn’t reactive friendly api, if you really need it, use extra thread like this:

        req.content()
                .as(InputStream.class)
                .forSingle(is -> executorService.submit(() -> {
                    try {
                        String msg = new String(is.readAllBytes());
                        LOG.log(Level.INFO, msg);
                        res.send(msg);
                    } catch (IOException e) {
                        res.send(e);
                    }
                }))
                .exceptionally(res::send);

But you can work it out reactively just with one thread(its waaay faster):

        req.content()
                .collect(StringBuilder::new, (sb, dataChunk) -> {
                    sb.append(new String(dataChunk.bytes()));
                    dataChunk.release(); //releasing the chunks is important to avoid memory leak as netty does ref counting with underlying ByteBuffs 
                })
                .map(StringBuilder::toString)
                .forSingle(msg -> {
                    LOG.log(Level.INFO, msg);
                    res.send(msg);
                })
                .exceptionally(res::send);