Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Handle accessibility incomplete and violations simultaneously

See original GitHub issue

FAQ

Yes, my issue is not about variability or throttling.
~~[ ] Yes, my issue is not about a specific accessibility audit (file with axe-core instead).~~

URL

https://accessibility-experiments-90d5b5.netlify.app/lighthouse-bypass-audit/index.html

What happened?

The bypass audit based on Axe’s bypass rule always shows as passing when it’s applicable, even when there are issues on the page that Axe itself detects.

Sample page: https://accessibility-experiments-90d5b5.netlify.app/lighthouse-bypass-audit/index.html

Lighthouse results for bypass on this page (JSON export)

"bypass": {
      "id": "bypass",
      "title": "The page contains a heading, skip link, or landmark region",
      "description": "Adding ways to bypass repetitive content lets keyboard users navigate the page more efficiently. [Learn more](https://web.dev/bypass/).",
      "score": 1,
      "scoreDisplayMode": "binary",
      "details": {
        "type": "table",
        "headings": [],
        "items": []
      }
    },

Axe CLI results for the same page

Command:

axe -r bypass https://accessibility-experiments-90d5b5.netlify.app/lighthouse-bypass-audit/index.html --save bypass.json && cat bypass.json

Output:

[
  {
    "testEngine": {
      "name": "axe-core",
      "version": "4.3.5"
    },
    "testRunner": {
      "name": "axe"
    },
    "testEnvironment": {
      "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/104.0.5112.101 Safari/537.36",
      "windowWidth": 800,
      "windowHeight": 600,
      "orientationAngle": 0,
      "orientationType": "landscape-primary"
    },
    "timestamp": "2022-09-03T10:13:28.251Z",
    "url": "https://accessibility-experiments-90d5b5.netlify.app/lighthouse-bypass-audit/index.html",
    "toolOptions": {
      "runOnly": {
        "type": "rule",
        "values": [
          "bypass"
        ]
      },
      "reporter": "v1"
    },
    "inapplicable": [],
    "passes": [],
    "incomplete": [
      {
        "id": "bypass",
        "impact": "serious",
        "tags": [
          "cat.keyboard",
          "wcag2a",
          "wcag241",
          "section508",
          "section508.22.o"
        ],
        "description": "Ensures each page has at least one mechanism for a user to bypass navigation and jump straight to the content",
        "help": "Page must have means to bypass repeated blocks",
        "helpUrl": "https://dequeuniversity.com/rules/axe/4.3/bypass?application=webdriverjs",
        "nodes": [
          {
            "any": [
              {
                "id": "internal-link-present",
                "data": null,
                "relatedNodes": [],
                "impact": "serious",
                "message": "No valid skip link found"
              },
              {
                "id": "header-present",
                "data": null,
                "relatedNodes": [],
                "impact": "serious",
                "message": "Page does not have a heading"
              },
              {
                "id": "landmark",
                "data": null,
                "relatedNodes": [],
                "impact": "serious",
                "message": "Page does not have a landmark region"
              }
            ],
            "all": [],
            "none": [],
            "impact": "serious",
            "html": "<html lang=\"en\" class=\"deque-axe-is-ready\">",
            "target": [
              "html"
            ],
            "failureSummary": "Fix any of the following:\n  No valid skip link found\n  Page does not have a heading\n  Page does not have a landmark region"
          }
        ]
      }
    ],
    "violations": []
  }
]

What did you expect?

Lighthouse’s results for bypass should be consistent with those of Axe. In this instance, Axe reports bypass as “incomplete”, so it shouldn’t be a pass with Lighthouse. I would have expected either a “Additional items to manually check”, or an error.

What have you tried?

I’ve investigated how the Axe rule works, and suspect the issue is due to Axe changing the output of this rule from violations to incomplete. This apparently happened in Axe v4.2 because although the rule can detect the absence of “bypass” functionality, it can’t detect whether there is actually anything to bypass on a page.

I suspect the issue in Lighthouse is in the Axe audit processing logic, around here, where Axe handles incomplete results differently based on the audit’s display mode:

https://github.com/GoogleChrome/lighthouse/blob/fd304ed182bcbf66bb4225e474be9799f14667b1/core/audits/accessibility/axe-audit.js#L47-L58

I’m unclear whether the current logic could work or whether there needs to be additions here.

How were you running Lighthouse?

Chrome DevTools, web.dev

Lighthouse Version

9.6.6

Chrome Version

107

Node Version

No response

OS

Mac

Relevant log output

No response

Issue Analytics

State:
Created a year ago
Comments:6 (3 by maintainers)

Top GitHub Comments

1reaction

adamrainecommented, Oct 7, 2022

Thanks for the research @thibaudcolas! I think should merge everything into this audit, but good to know which audits this problem affects.

1reaction

adamrainecommented, Sep 6, 2022

Thanks for reporting this and doing an investigation! Not very often that someone reports a page that should be failing an audit but isn’t.

Your analysis is correct. The failure is hidden by this line: https://github.com/GoogleChrome/lighthouse/blob/fd304ed182bcbf66bb4225e474be9799f14667b1/core/audits/accessibility/axe-audit.js#L58

The audit is not classified as informative so we ignore any incomplete reports from axe. I think it’s worth discussing making the bypass audit informative to surface these cases.