Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

JsonGzipHttpClient causing back-end app to throw "unexpected end of file" error

See original GitHub issue

Describe the bug

I finally got around to testing the JsonGzipHttpClient that was added to the Sink. While testing, I found a bug when sending logs to Fluentd. When Fluentd tries to decompress the payload it outputs the following error: 2021-05-10 02:04:17 +0000 [warn]: #0 [in_http] fails to decode payload error="unexpected end of file". After some investigating, I found that when the gzip stream is closed/disposed, there are an additional 4 bytes that are added to stream. However, when sending the payload before the stream is closed/disposed, the 4 bytes are missing. It’s due to these four bytes, that the Fluentd agent is unable to decompress payload.

To Reproduce Steps to reproduce the behavior:

Here is the code that works. Screenshot 1 shows the length of the content payload for this code


            // Create byte stream 
            var buffer = ConvertToBytes($"[ {jsonValue1}, {jsonValue2} ]");
            using var input = new MemoryStream(buffer);

            // Convert byte stream into Gzip Stream
            using var output = new MemoryStream();
            using (var gzipStream = new GZipStream(output, CompressionLevel.Fastest))
            {
                input.CopyTo(gzipStream);
            }

            // Create the Http Content stream w/headers
            using var contentStream = new MemoryStream(output.ToArray());
            var content = new StreamContent(contentStream);

            content.Headers.Add("Content-Type", "application/json");
            content.Headers.Add("Content-Encoding", "gzip");

            // Http post method
            using var client = new HttpClient();
            var response = client.PostAsync(content: content, requestUri: requestUri).Result;

            Console.WriteLine(response.IsSuccessStatusCode);

Here is the code that causes an issue. Screenshot 2 shows the length of the content payload for this code

            // Create byte stream 
            var buffer = ConvertToBytes($"[ {jsonValue1}, {jsonValue2} ]");
            using var input = new MemoryStream(buffer);

            using var output = new MemoryStream();
            using var gzipStream = new GZipStream(output, CompressionLevel.Fastest);

            input
                .CopyToAsync(gzipStream)
                .Wait();
            gzipStream.FlushAsync();

            output.Position = 0;

            var content = new StreamContent(output);
            content.Headers.Add("Content-Type", "application/json");
            content.Headers.Add("Content-Encoding", "gzip");

            // Http post method
            using var client = new HttpClient();
            var response = client.PostAsync(content: content, requestUri: requestUri).Result;

            Console.WriteLine(response.IsSuccessStatusCode);

Expected behavior The logs should be sent over Http compressed and the receiving application should be able to decompress without any issues.

Screenshots If applicable, add screenshots to help explain your problem.

Length of the Payload that works. Screenshot 1