Bug: Url host name parsing result is incorrect when hostname contains unicode character

Bug Report

Description

Url host name parsing result is incorrect when hostname contains unicode

Steps to Reproduce

var url1 = new Url("http://ec².com");
var url2 = new Url("http://a.com/ec²");

Console.WriteLine(url1.Href);
Console.WriteLine(url2.Href);

Expected output:

http://ec².com/
(or maybe http://ec2.com/ which will be the final host name sent for dns lookup)
http://a.com/ec%C2%B2

Actual output:

http://ec.com/
http://a.com/ec%C2%B2

Note that ² is missing.

Environment details: Windows 10 .Net Framework 4.6.2

Possible Solution

Problem only exist in hostname part

I saw comment:

//TODO finish with
//https://url.spec.whatwg.org/#concept-host-parser

I guess host name with Unicode characters is not implemented yet. Do you have any plan on that or do you need help on that?

[hyspace]

Please reopen this issue.

After reading the host-parsing part of spec

I found the current implementation still have a problem.

Consider this example input:

http://www.exampl？e.com/

Spec will consider this host name as invalid but current Url will think it is valid.

The problem is on https://github.com/AngleSharp/AngleSharp/blob/devel/src/AngleSharp/Url.cs#L1062 Current TrySanatizeHost function is doing a 2-passes validation of host name, while in the spec it is a 3 passes process:

1. 3.5.4 string percent decoding
2. 3.5.5 domain to ASCII
3. 3.5.7 check forbidden host code point

For example above, the result of step 2 may contains forbidden host code point. ？ will be mapped to ? and then should cause parse failure.

If you don’t have concern, I would like to implement the correct domain parsing including the ipv4 and ipv6 part.

[hyspace]

This plan sounds good to me now. I will still spend some time on the URL standard and IdnMapping Class to see if there are any potential issue.

Thanks!