Gzip encoding

Bug Report

Prerequisites

• [Y ] Can you reproduce the problem in a MWE?
• [ Y] Are you running the latest version of AngleSharp?
• [?] Did you check the FAQs to see if that helps you?
• [Y] Are you reporting to the correct repository? (there are multiple AngleSharp libraries, e.g., AngleSharp.Css for CSS support)
• [Y] Did you perform a search in the issues?

For more information, see the CONTRIBUTING guide.

Description

I’m seeing the same issue as #416 - downloaded page is gzip encoded and AngleSharp is not decompressing it.

Steps to Reproduce

Does not work:

var config = Configuration.Default.WithLocaleBasedEncoding().WithDefaultLoader();
var address = "https://www.powerball.com";
var context = BrowsingContext.New(config);
var document = await context.OpenAsync(address);

Expected behavior: [What you expected to happen]

Document should be decompressed plain text.

Actual behavior:

Document is “garbled” gzipped content.

Environment details: [OS, .NET Runtime, …]

Windows 11 x64, .NET 7.0.302, AngleSharp 1.0.3

Possible Solution

Works:

HttpClientHandler handler = new HttpClientHandler()
{
    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};        
var http = new HttpClient(handler);
var body = await http.GetStringAsync("https://www.powerball.com");

var config = Configuration.Default;
var context = BrowsingContext.New(config);
var document = await context.OpenAsync(req => req.Content(body));

Working around this by using HTTP Client. Is this the recommended approach? I searched for “org:AngleSharp gzip” and didn’t see any recommendations or FAQ guidance. I assume this should be out-of-box automatic behavior, so maybe I’m missing something.

[FlorianRappl]

By the way I could finally reproduce this. The server is sometimes returning Brotli (“br”) compressed responses - even though the Accept-Encoding header tells it that only “deflate” and “gzip” are supported.

In such case we’ll now throw an exception. The response would have been gibberish anyway and this way one can react. Should be a rare case though - this is certainly a problem on the webserver.

[FlorianRappl]

Working around this by using HTTP Client. Is this the recommended approach?

Yes, the requester coming with AngleSharp is not using the HttpClient and should only be used in simple cases. If you heavily rely on IO then you should use AngleSharp.Io.

I’ll see if this is a general problem with the requester (not being able to process gzip) or if this is something with the page. If its a general problem then we need to drop the “gzip” from accepted encodings.