Are CSP audits too strict?

Feature request summary

The csp-xss audits added in Lighthouse v8 are quite strict and quite opinionated. While they do expose best practices, some of the options are advanced CSP configurations and I think a lot of sites will struggle to pass these audits.

I have always found Lighthouse strikes the balance well between pushing for best practices versus realistic, achievable targets for the average website (with a little effort), but I fear the balance here is off, and the worry is that many people will just ignore this audit as “unachievable” and/or turn it off in any automated Lighthouse runs rather than using this to improve their security - which I presume is the intention of this audit. Or worse still rush into a complex topic like CSP without fully understanding it (more on this later). As I say to me Lighthouse audits should be tough but achievable.

Having strict settings on a specific tool like the CSP evaluator (that is being used as the basis for this audit) is fine and to be expected. But having this on a more generic tool like Lighthouse, may need a different thought process to make it more palatable to the wider audience who may not be security experts.

In my opinion we should not push too hard for advanced configurations to be adopted by those that fail to understand what it means. We know that not enough sites use a CSP at all, never mind some of these advanced options, and I think initially we should aim to push, and reward, getting basic CSPs in place rather than set the bar too high and insisting on an advanced CSP.

I appreciate that the CSP Audit is not yet included in the Best Practice score, but I think some of these settings should be reconsidered before that happens.

In particular, I think the following could perhaps be considered to be Warnings rather than High for now and not fail the audit for missing these:

• strictDynamic - using hashes and nonces are advanced techniques that require server side set up to send them. So simple blogs and many simple hosting providers and CMS’s and tools cannot support these. Based on last year’s Web Almanac data 0.225% of mobile websites used strict-dynamic. While they do offer the best protection, and I agree this should be flagged to users, I personally think failing the audit for not using these is a step too far given current adoption of this, and should be instead considered for future, with just a warning for now. Personally I think that would drive adoption better than jumping straight to a full fail now.
• missingObjectSrc - The name of this audit is misleading is it also flags when the object-src is provided but not set to 'none'. Perhaps there should be two audits: 1) if missing (High - so fail) and 2) if present but set to anything but none (Medium - so warning but not fail, as some sites explicitly choose to allow object src, e.g. to have interactive SVGs).

I also wonder if this setting is too much to push now:

• reportingDestinationMissing - personally I find CSP reports far too noisy to really be useful (mostly due to lots and lots of bad extensions), but that’s a different issue. Again I think this is an advanced setting IMHO. However since it’s currently a Medium rather than High and (as I understand it) won’t fail the audit, I can live with this one. Personally I’d make it an Info one though.

I’m one of the HTTP Archive maintainers, and happy to get some more stats here if you want. The July HTTP Archive crawl is due to finish in the next week and it should have been run with v8.0.0, but I suspect very few will be passing which to me makes this audit too strict, at least until adoption is driven up.

What is the motivation or use case for changing this?

Keep audits realistic and achievable, but with warnings and suggestions for improvements to drive forward best practices, rather than mandating it too early.

This will also reduce the chance of breaking sites with inexperienced developers jumping into complex topics like CSP and going straight to advanced settings without necessarily understanding them (see past history with HSTS preload or HPKP).

How is this beneficial to Lighthouse?

Stops users giving up on audits as unachievable and losing confidence in Lighthouse as a useful tool.

[connorjclark]

It’s the inspector-issues, which uses a new protocol where Chrome will tell us about certain issues … and we mistakenly accept any error there as worthy of failing the inspector-issues audit. Certainly not our intention 😃

Team, we could

1. make inspector-issues 0 weight
2. ignore “CSP” for calculating failure

I’m leaning towards 1) but I don’t have a strong argument right now.

Incidentally, the note about report-uri in that link I gave doesn’t quite square with the advice in: developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/report-uri

@adamraine do you recall why we aren’t suggesting report-to here? Probably should be doing both.

EDIT actually might it be working as intended because the CSP is blocking resources and chrome is pointing that out? Maybe we just need to show the actual error message in the report.

[patrickhulce]

Thanks for weighing in with important feedback folks!

We’ve decided to make a few changes:

• We will adopt the missingObjectSrc/object-src is not none changes.
• We will relax the reporting requirement.
• We will still enforce strictDynamic but the audit will not be scored (turn red) until there is plentiful ecosystem support and adoption has reached at least 2%.