Incorrect main document used when retrieving page load errors

Provide the steps to reproduce

1. Run LH programmatically on page that redirects a similarly matching URL (e.g. same startsWith(request.url)), where the redirect response does not return text/html as its mime type.

More Information

I’m not sure how common this for non-authenticated redirects, but in our case we are running LH on authenticated pages using cookie-based login (similar to this doc, but with Node API).

The redirect response when hitting the requestedUrl (prior to auth) does not return HTML, and so the content-type response header is empty. It then redirects to the authentication server, and finally back to the originally requested document this time with the correct mime type text/html.

What is the current behavior?

LH reports error NOT_HTML even though document loaded in the final URL after redirecting is text/html.

I don’t see any documentation on this, but according @brendankenny in #11430, it should be the final matching URL (that is an html document) when loading the page.

The error occurred starting in version 6.2.0 of LH from #11042.

I have a fix locally by using the last matching HTML document (as opposed to the first) in findMainDocument(records, finalURL).

A naive one-line fix that works on my site in test is:

// should probably search backwards using loop
const mainResource = records.slice().reverse().find(request => finalURL.startsWith(request.url) &&
        URL.equalWithExcludedFragments(request.url, finalURL));

As for the logic when finalUrl is not supplied, I think the comment on line 452 still applies.

What is the expected behavior?

LH doesn’t fail when valid redirect eventually responds with HTML on a matching finalUrl.

Environment Information

• Affected Channels: Node
• Lighthouse version: >=6.2.0
• Chrome version: N/A
• Node.js version: 12.9.2
• Operating System: MacOS (and Docker Linux Container)

Related issues

• #11042
• #11430

[bradfrosty]

Hey @patrickhulce, I should have a PR up soon. All good from my end to contribute and CLA has been signed.

[bradfrosty]

@patrickhulce sorry for the delay. Figuring out the legal approval on my end before signing the CLA and submitting a contribution 🙂