Lighthouse cannot get a Performance score ~30% of the time

Hey folks! At SpeedCurve we’re running Lighthouse about 100,000 times every day across almost as many distinct URLs. We consider a LH run to have failed if it doesn’t return a Performance score. We have a baseline failure rate of 8-9% (mostly due to bot managers) but we noticed that around April 25 the failure rate jumped to about 30%.

The raw data for this chart is in a public spreadsheet if you want to dive into it more. You’ll notice the failure rate is less for mobile than it is for desktop.

We don’t track the failure reason, but anecdotally the NO_SCREENSHOTS error is pretty common. Here’s an example test for a URL that gets this error roughly 1 in 3 runs. I didn’t see this error often before 25 April.

Environment Information

SpeedCurve runs its Lighthouse testing via WebPageTest. WPT performs its own throttling, using tc-netem for the network and Chrome DevTools for the CPU.

• Affected Channels: CLI
• Lighthouse version: 5.1.0
• Node.js version: 10.x
• Operating System: Ubuntu 18.04

Example command that would be run by wptagent:

# {PORT} is the port of an already-running Chrome instance, potentially with CPU throttling applied.
lighthouse "{URL}" --disable-network-throttling --disable-cpu-throttling --throttling-method provided --enable-error-reporting --max-wait-for-load 60000 --port {PORT} --output html --output json --output-path "{PATH}"

[zeman]

@brendankenny For more context the WebPageTest agents reboot every hour and update the browsers and Lighthouse to latest release/stable version and that’s what we test with.

[wildlyinaccurate]

It looks like Chrome 75 has returned our error rate back to normal. I’ll close this ticket for now but I’m still more than happy to assist debugging Chrome 74 if that’s useful for the Lighthouse team.