FullHttpRequest doesn't return proper content anymore

It is server code. Server receives HTTP POST request, which contains body content. Server converts the request to FullHttpRequestand gets its content as a buffer. For some unknown reason buffer became not readable in versions 4.1.64.Final and 4.1.65.Final, but up to version 4.1.63.Final everything worked just fine. Pipeline contains just HttpServerCodec and HttpObjectAggregator.

Expected behavior

FullHttpRequest of Multipart POST request returns proper content of the body.

Actual behavior

FullHttpRequest of Multipart POST request doesn’t return proper content of the body.

Steps to reproduce

Try to read content of FullHttpRequest and get it as a String.

Minimal yet complete reproducer code (or URL to code)

if (req instanceof FullHttpRequest) {
    final FullHttpRequest fhr = (FullHttpRequest) req;
    // Actually contains data, but is not ready for reading anymore
    final ByteBuf bb = fhr.content();
    // String is empty now, but was ok in versions <= 4.1.63.Final
    final String s = bb.toString(StandardCharsets.UTF_8);
}

Netty version

Works ok up to 4.1.63.Final, but doesn’t work anymore in 4.1.64.Final and 4.1.65.Final

JVM version (e.g. java -version)

openjdk version “16” 2021-03-16 OpenJDK Runtime Environment (build 16+36-2231) OpenJDK 64-Bit Server VM (build 16+36-2231, mixed mode, sharing)

OS version (e.g. uname -a)

Linux x86_64 x86_64 x86_64 GNU/Linux and Windows 10

[bxqgit]

@fredericBregier Thanks for the detailed answer! 👍🏻

I want to use HttpObjectAgregator and FullHttpRequest to keep the code simple, even though it may not be the most efficient. Sometimes I think that Netty is too low level for my needs. 😄 But since everything works just fine, I have no reason to rewrite code to some other high level framework. 🤷‍♂️

I want to extract all the possible representations of the POST data. If it is a form, then I want to have key -> value pairs. If it is just a simple plain content, I want to have it as a String.

So I just changed the order. First I get the content from FullHttpRequest and then I use HttpPostRequestDecoder, and now I don’t care if it modifies the request or not. 😄

[fredericBregier]

@bxqgit OK, I understand.

So you are indeed using a Multipart decoding. I have found several issues in the way you using it:

• In general, it is not recommended to use the HttpObjectAgregator in your pipeline when facing Multipart. The reason is that it will fully load everything in memory. But as you’re using a Multipart, it might be “too much” for the JVM. So you might change this as follow:

  • First, remove the HttpObjectAgregator from your pipeline

  • Then adapt your code as follow (inspired from https://github.com/netty/netty/blob/4.1/example/src/main/java/io/netty/example/http/upload/HttpUploadServerHandler.java )

    // The factory in general is initialized once
    private static final HttpDataFactory factory =
        new DefaultHttpDataFactory(DefaultHttpDataFactory.MINSIZE); // Disk if size exceed
    
    // Store the decoder between each and every "chunk" for the same request
    private HttpPostRequestDecoder decoder;
    
    @Override
    public void channelRead0(ChannelHandlerContext ctx, HttpObject msg) throws Exception {
        if (msg instanceof HttpRequest) {
            HttpRequest request = (HttpRequest) msg;
            try {
                decoder = new HttpPostRequestDecoder(factory, request);
            } catch (ErrorDataDecoderException e1) {
                // Your error handling
                return;
            }
        }
    
        // check if the decoder was constructed before (if not, could be an issue)
        if (decoder != null) {
            if (msg instanceof HttpContent) {
                // New chunk is received
                HttpContent chunk = (HttpContent) msg;
                try {
                    decoder.offer(chunk);
                } catch (ErrorDataDecoderException e1) {
                    // Your error handling
                    return;
                }
            }
            // check for end of multipart (which is also an HttpContent)
            if (chunk instanceof LastHttpContent) {
                // finaly use all data (may be use before the end,
                // but keep in mind that not all data are there, so up to your business logic)
                readHttpDataChunkByChunk()
                // Finalize the decoder
                reset();
            }
        }
    }
    
    private void reset() {
        // destroy the decoder to release all resources
        decoder.destroy();
        decoder = null;
    }
    
    /**
     * Example of reading request by chunk and getting values from chunk to chunk
     */
    private void readHttpDataChunkByChunk() {
        try {
            while (decoder.hasNext()) {
                InterfaceHttpData data = decoder.next();
                if (data != null) {
                    // new value to use in your way
                    useTheNewData(data);
                    data.release();
                }
            }
        } catch (EndOfDataDecoderException e1) {
            // end
        }
    }
    

• Now to your issue:

  • Whatever using onely one FullHttpRequest or using multiple HttpContent (when chunked, the first item is a HttpRequest without any content, followed by multiple HttpContent, ending with a last LastHttpContent that closes the chunk reception)
  • When you offer the chunks to the decoder, the decoder consumes each and every chunk or the only one FullHttpRequest if not chunked
  • This means that the request is now fully read, and therefore the ByteBuf has its readerIndex sets to writerInder.
  • So the reason you cannot read it anymore (except if you reset readerIndex to 0 as you did).
  • The change occurs since previously the decoder did not act as needed. It should read all buffers, and not keeping them unchanged (indeed, it was really depending on how the chunks arrived, so not consistent). All decoders reads the input ByteBufs and therefore the HttpRequest is not immutable regarding its content.

So it is not an issue but a normal way. You’re not using a decoder correctly.

I don’t know why you want to get the “full” content of a Multipart request into a String, but if so, you can build it through the HttpData retrieves from the decoder, as follow:

    /**
     * NEW: add a StringBuilder that you can define as attached to the session and initialize
     * and destroyed for each and every HttpRequest as the decoder
     */
    private StringBuilder builder = null;
    @Override
    public void channelRead0(ChannelHandlerContext ctx, HttpObject msg) throws Exception {
        if (msg instanceof HttpRequest) {
            HttpRequest request = (HttpRequest) msg;
            try {
                decoder = new HttpPostRequestDecoder(factory, request);
            } catch (ErrorDataDecoderException e1) {
                // Your error handling
                return;
            }
            // NEW: initialize the StringBuilder
            builder = new StringBuilder();
        }
        ...
            // check for end of multipart (which is also an HttpContent)
            if (chunk instanceof LastHttpContent) {
                // finaly use all data (may be use before the end,
                // but keep in mind that not all data are there, so up to your business logic)
                readHttpDataChunkByChunk()
                // NEW: Use your StringBuilder
                final String s = builder.toString();
                // Finalize the decoder
                reset();
            }
        }
    }

    /**
     * Example of reading request by chunk and getting values from chunk to chunk
     */
    private void readHttpDataChunkByChunk() {
        try {
            while (decoder.hasNext()) {
                InterfaceHttpData data = decoder.next();
                if (data != null) {
                    // new value to use in your way
                    useTheNewData(data);
                    // NEW: Change here: use the StringBuilder
                    builder.append(data.getString(UTF_8));
                    data.release();
                }
            }
        } catch (EndOfDataDecoderException e1) {
            // end
        }
    }
    
    private void reset() {
        // destroy the decoder to release all resources
        decoder.destroy();
        decoder = null;
        // NEW: StringBuilder clean
        builder = null;
    }

Hoping this is clear enough to help you.