Programmatic performance scores are consistently worse than from DevTools

For some URLs, but not others, I’m getting consistently lower performance scores when running Lighthouse programmatically than from Chrome DevTools.

For example:

https://marksandspencer.com
Programmatic: 64, 64, 63, 64, 55
DevTools: 86, 73, 84, 77, 80

https://www.littlewoods.com
Programmatic: 46, 48, 50, 50, 49
DevTools: 60, 59, 57, 57, 60

https://asos.com/emporio-armani/emporio-armani-logo-slippers-in-navy/prd/10336705
Programmatic: 10, 8, 10, 9, 8
DevTools: 22, 25, 18, 17, 23

By comparison, for many other sites I get quite consistent results:

https://www.houseoffraser.co.uk
Programmatic: 20, 22, 22, 21, 18
DevTools: 20, 24, 23, 21, 21

https://www.johnlewis.com
Programmatic: 55, 58, 54, 58, 57
DevTools: 56, 53, 52, 52, 52

For the programmatic audit I’m running the Lighthouse sample code with Node 10.6.0.

For the DevTools audit I’m using Canary 73.0.3659.0. AFAIK I don’t have any extensions installed that could affect Lighthouse.

I’m using a Mac, and I’ve tried on home wifi (around 30 Mbps) and work wifi (>100 Mbps).

FWIW I’ve tried headless and non-headless — as expected, that doesn’t seem to make any difference.

[patrickhulce]

Thanks for the reminder @Hoten! I think he was using Canary to start with 👍

Since this is WAI in Lighthouse, I’m going to go ahead and close, but feel free to chime in with more examples if they don’t seem to fall into one of these categories!

[connorjclark]

Please note that there is currently a bug in the latest stable Chrome DevTools that results in Clear Storage never happening - https://github.com/GoogleChrome/lighthouse/issues/6772

This is fixed in Canary and will be in Chrome 72.