WPT started getting a lot of PROTOCOL_TIMEOUT errors

FAQ

• Yes, my issue is not about variability or throttling.
• Yes, my issue is not about a specific accessibility audit (file with axe-core instead).

URL

https://www.catchpoint.com/

What happened?

Somewhere last week we started seeing a lot of failures on WebPageTest when running lighthouse audits. Not sure if it is a Chrome change or Lighthouse change since the Agents auto-update both.

The logs are all failing with PROTOCOL_TIMEOUT, usually when issuing an Emulation command of some kind. In this example it was trying to disable JS on a hung page but there have been other cases where it happens after collecting the trace and when it is resetting the network emulation (throttling method is dev tools with no network throttling but a CPU throttle).

What did you expect?

I expected it to run the test and collect the audits.

What have you tried?

It seems to only happen when dev tools CPU throttling is enabled. If I switch to “provided” or don’t provide a CPU throttle then the tests run fine.

Best guess is that the emulation commands are run in parallel with other commands (like collecting the trace) that take longer when CPU throttling is enabled and something pushed it over the edge where now it is timing out.

I’m rolling out verbose logging now so I should have better logs shortly that should show what else is going on when the errors occur.

How were you running Lighthouse?

CLI, WebPageTest

Lighthouse Version

8.1.0

Chrome Version

92.0.4515.107

Node Version

12.22.2

Relevant log output

lighthouse "https://www.catchpoint.com/" --channel wpt --enable-error-reporting --max-wait-for-load 80000 --port 9238 --output html --output json --output-path "/home/ubuntu/wptagent/work/wpt_use1_instance-10.10.1.227/210728_AiDcNZ_496f5610778428860fd9464f1ffff300.1.0/lighthouse.json" --throttling-method devtools --throttling.requestLatencyMs 0 --throttling.downloadThroughputKbps 10000 --throttling.uploadThroughputKbps 10000 -throttling.cpuSlowdownMultiplier 7.912 --save-assets --skip-audits screenshot-thumbnails,final-screenshot --chrome-flags="--user-agent='Mozilla/5.0 (Linux; Android 6.0.1; Moto G (4)) AppleWebKit/537.36 (KHTML like Gecko) Chrome/86.0.4240.110 Mobile Safari/537.36 PTST/210728.223249'"
Wed, 28 Jul 2021 23:14:53 GMT LH:status Connecting to browser
Wed, 28 Jul 2021 23:14:53 GMT LH:status Resetting state with about:blank
Wed, 28 Jul 2021 23:14:53 GMT LH:status Benchmarking machine
Wed, 28 Jul 2021 23:14:54 GMT LH:status Initializing…
Wed, 28 Jul 2021 23:14:54 GMT LH:status Running defaultPass pass CSSUsage, JsUsage, ViewportDimensions, ConsoleMessages, AnchorElements, ImageElements, LinkElements, MetaElements, ScriptElements, MainDocumentContent, GatherContext, GlobalListeners, AppCacheManifest, Doctype, DOMStats, OptimizedImages, PasswordInputsWithPreventedPaste, ResponseCompression, TagsBlockingFirstPaint, FontSize, EmbeddedContent, RobotsTxt, TapTargets, Accessibility, TraceElements, InspectorIssues, SourceMaps
Wed, 28 Jul 2021 23:14:54 GMT LH:status Resetting state with about:blank
Wed, 28 Jul 2021 23:14:54 GMT LH:status Cleaning browser cache
Wed, 28 Jul 2021 23:14:54 GMT LH:status Preparing network conditions
Wed, 28 Jul 2021 23:14:54 GMT LH:status Beginning devtoolsLog and trace
Wed, 28 Jul 2021 23:14:54 GMT LH:status Loading page & waiting for onload
Wed, 28 Jul 2021 23:16:14 GMT LH:waitFor:warn Timed out waiting for page load. Checking if page is hung...
Wed, 28 Jul 2021 23:16:15 GMT LH:waitFor:warn Page appears to be hung, killing JavaScript...
Wed, 28 Jul 2021 23:16:45 GMT LH:status Disconnecting from browser...
Runtime error encountered: Waiting for DevTools protocol response has exceeded the allotted time. (Method: Emulation.setScriptExecutionDisabled)
LHError: PROTOCOL_TIMEOUT
    at Timeout.<anonymous> (/usr/lib/node_modules/lighthouse/lighthouse-core/gather/driver.js:322:21)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7)

[patrickhulce]

Huzzah! It’s the CPU profiler and we don’t even need to use it right now! 🎉

[pmeenan]

If it helps, I’m happy to provide access to the VM to debug. If you send me a public SSH key I can install it on the VM and provide information on how to debug in the WPT agent config