DevTools webtests improvements

Summary

Now that we’ve started to bring the DevTools testing into core we can start to make them even stronger 😃

Miscellaneous thoughts:

• Move LighthouseTestRunner into core as well so we can make changes to the shared utilities.
• Strengthen audit assertions, for example we had to disable all perf assertions in the past because screenshots were flaky. It would be awesome if we could flip the script to get Chromium’s perspective to consider flaky screenshot collection a bug and fix that instead of disabling the tests that point it out 😃
• Assertions on the rendering of the report. AFAIK all our current tests assert on the LHR are obtained from a spy on internal methods. Some sort of testing that the complete rendering pipeline works would be great for testing PRs like #11411
• A package.json script that would build and open the latest Lighthouse in DevTools for manual inspection.
• fix caching TTL https://github.com/GoogleChrome/lighthouse/issues/11414#issuecomment-691294975
• remove the TestExpectations that marks one of the tests flaky (probably not flaky anymore)
• startup errors should show error and not timeout https://github.com/GoogleChrome/lighthouse/pull/10630#issuecomment-694400741
• only put explanation in snapshots when audit fails
• don’t hide output of http server (blocked port hangs)
• git pull devtools checkout in .tmp #11717 #11678

[brendankenny]

wdyt about dropping the need for wget?

Sucks to have to install it so some scripts can basically curl some things 😃

I don’t see any unusual things going on, seems like it would be a simple replacement?

[patrickhulce]

wait, but why didn’t #11411 break the renderer since it gets rid of Runtime.cachedResources and uses globalThis.EXPORTED_CACHED_RESOURCES_ONLY_FOR_LIGHTHOUSE but the cached devtools code presumably wouldn’t have that new object yet?

I don’t know. At the time I said that I hadn’t looked at #11417 yet 😆 I had it in my head that we wouldn’t have rebuilt for devtools at all if the hash key wasn’t changed (like how the lantern tests work), but now I’m seeing it should’ve been run with a mismatched old DevTools, so I have the same question as you.