axe bugs in LH 6.0 (null accessibility score)

We are using Lighthouse 6 programmatically using node module to get performance, accessibility and SEO scores. For few URLs listed below, it returns null accessibility scores when tested for desktop with no apparent errors. Lighthouse v5 worked fine for us and we have been using it for less than 2 years.

It does return accessibility scores for some URLs such as google.com, abc.com etc.

Provide the steps to reproduce

1. Run Lighthouse 6.0.0 with node module “lighthouse” and “chrome-launcher” OR “puppeteer” for below URLs

• https://quickbooks.intuit.com
• https://quickbooks.intuit.com/pricing
• https://quickbooks.intuit.com/online/
• https://quickbooks.intuit.com/oa/get-quickbooks/

1. Check Results object accessibility score is null. Example:

"accessibility": {
    "title": "Accessibility",
    "description": "These checks highlight opportunities to [improve the accessibility of your web app](https://developers.google.com/web/fundamentals/accessibility). Only a subset of accessibility issues can be automatically detected so manual testing is also encouraged.",
    "manualDescription": "These items address areas which an automated testing tool cannot cover. Learn more in our guide on [conducting an accessibility review](https://developers.google.com/web/fundamentals/accessibility/how-to-review).",
    "auditRefs": [
      {
        "id": "accesskeys",
        "weight": 0,
        "group": "a11y-navigation"
      },
      {
        "id": "aria-allowed-attr",
        "weight": 10,
        "group": "a11y-aria"
      },
      {
        "id": "aria-hidden-body",
        "weight": 10,
        "group": "a11y-aria"
      },
      {
        "id": "aria-hidden-focus",
        "weight": 3,
        "group": "a11y-aria"
      },
      {
        "id": "aria-input-field-name",
        "weight": 0,
        "group": "a11y-aria"
      },
      {
        "id": "aria-required-attr",
        "weight": 10,
        "group": "a11y-aria"
      },
      {
        "id": "aria-required-children",
        "weight": 10,
        "group": "a11y-aria"
      },
      {
        "id": "aria-required-parent",
        "weight": 10,
        "group": "a11y-aria"
      },
      {
        "id": "aria-roles",
        "weight": 10,
        "group": "a11y-aria"
      },
      {
        "id": "aria-toggle-field-name",
        "weight": 0,
        "group": "a11y-aria"
      },
      {
        "id": "aria-valid-attr-value",
        "weight": 10,
        "group": "a11y-aria"
      },
      {
        "id": "aria-valid-attr",
        "weight": 10,
        "group": "a11y-aria"
      },
      {
        "id": "button-name",
        "weight": 10,
        "group": "a11y-names-labels"
      },
      {
        "id": "bypass",
        "weight": 3,
        "group": "a11y-navigation"
      },
      {
        "id": "color-contrast",
        "weight": 3,
        "group": "a11y-color-contrast"
      },
      {
        "id": "definition-list",
        "weight": 0,
        "group": "a11y-tables-lists"
      },
      {
        "id": "dlitem",
        "weight": 0,
        "group": "a11y-tables-lists"
      },
      {
        "id": "document-title",
        "weight": 3,
        "group": "a11y-names-labels"
      },
      {
        "id": "duplicate-id-active",
        "weight": 3,
        "group": "a11y-navigation"
      },
      {
        "id": "duplicate-id-aria",
        "weight": 10,
        "group": "a11y-aria"
      },
      {
        "id": "form-field-multiple-labels",
        "weight": 0,
        "group": "a11y-names-labels"
      },
      {
        "id": "frame-title",
        "weight": 3,
        "group": "a11y-names-labels"
      },
      {
        "id": "heading-order",
        "weight": 2,
        "group": "a11y-navigation"
      },
      {
        "id": "html-has-lang",
        "weight": 3,
        "group": "a11y-language"
      },
      {
        "id": "html-lang-valid",
        "weight": 3,
        "group": "a11y-language"
      },
      {
        "id": "image-alt",
        "weight": 10,
        "group": "a11y-names-labels"
      },
      {
        "id": "input-image-alt",
        "weight": 0,
        "group": "a11y-names-labels"
      },
      {
        "id": "label",
        "weight": 10,
        "group": "a11y-names-labels"
      },
      {
        "id": "layout-table",
        "weight": 0,
        "group": "a11y-tables-lists"
      },
      {
        "id": "link-name",
        "weight": 3,
        "group": "a11y-names-labels"
      },
      {
        "id": "list",
        "weight": 3,
        "group": "a11y-tables-lists"
      },
      {
        "id": "listitem",
        "weight": 3,
        "group": "a11y-tables-lists"
      },
      {
        "id": "meta-refresh",
        "weight": 0,
        "group": "a11y-best-practices"
      },
      {
        "id": "meta-viewport",
        "weight": 10,
        "group": "a11y-best-practices"
      },
      {
        "id": "object-alt",
        "weight": 0,
        "group": "a11y-names-labels"
      },
      {
        "id": "tabindex",
        "weight": 3,
        "group": "a11y-navigation"
      },
      {
        "id": "td-headers-attr",
        "weight": 3,
        "group": "a11y-tables-lists"
      },
      {
        "id": "th-has-data-cells",
        "weight": 3,
        "group": "a11y-tables-lists"
      },
      {
        "id": "valid-lang",
        "weight": 0,
        "group": "a11y-language"
      },
      {
        "id": "video-caption",
        "weight": 0,
        "group": "a11y-audio-video"
      },
      {
        "id": "video-description",
        "weight": 0,
        "group": "a11y-audio-video"
      },
      {
        "id": "logical-tab-order",
        "weight": 0
      },
      {
        "id": "focusable-controls",
        "weight": 0
      },
      {
        "id": "interactive-element-affordance",
        "weight": 0
      },
      {
        "id": "managed-focus",
        "weight": 0
      },
      {
        "id": "focus-traps",
        "weight": 0
      },
      {
        "id": "custom-controls-labels",
        "weight": 0
      },
      {
        "id": "custom-controls-roles",
        "weight": 0
      },
      {
        "id": "visual-order-follows-dom",
        "weight": 0
      },
      {
        "id": "offscreen-content-hidden",
        "weight": 0
      },
      {
        "id": "use-landmarks",
        "weight": 0
      }
    ],
    "id": "accessibility",
    "score": null
  }

What is the current behavior?

Accessibility score is null for desktop device for bunch of URLs when analyzed through node module of LH 6

What is the expected behavior?

Accessibility score should be between 0 to 1.

Environment Information

• Affected Channels: Node
• Lighthouse version: 6.0.0
• Chrome version: chrome-launcher ^0.13.2
• Node.js version: 10.20.1
• Operating System: Mac OS High Sierra

Related issues

[patrickhulce]

Thanks for digging in @spacedawwwg! That’s captured by https://github.com/dequelabs/axe-core/issues/2262 and we’ve requested another 3.5.x release to fix it sooner in @paulirish 's issue.

[patrickhulce]

@kaspergrubbe that’s a separate axe bug that hasn’t been fixed (see https://github.com/dequelabs/axe-core/issues/2483) this issue was for the set of bugs fixed in 3.5.5