Accessibility: expose element reference

Steps to reproduce

While using the puppeteer api page.accessibility.snapshot() the AXNode model leaves no trace of the DOM implementation. The underlying calls to CDP return nodeId (and backendDOMNodeId).

I suggest one or more of the following (will update if more ideas come in comments):

1. Give AXNode a new method ‘asElement’ which will create an ElementHandle from the stored nodeId or remoteObject reference.
2. Expose the nodeId or remoteObjectId on the AXNode (so it can be passed to other CDP calls).
3. Expose AXNode so CDP calls to Accessibility.getPartialAXTree can use this code to normalize the result the way snapshot() does.

• Puppeteer version: 1.10+
• Platform / OS version: any
• URLs (if applicable):
• Node.js version: latest

Please include code that reproduces the issue.

// ... open google.com
const tree = await this.page.accessibility.snapshot();
console.log(tree);

Yields:

{ role: 'WebArea',
  name: 'Google',
  children:
   [ { role: 'button', name: 'Dismiss' },
     { role: 'link', name: 'About' },
     { role: 'link', name: 'Store' },
     { role: 'link', name: 'Gmail' },
     { role: 'link', name: 'Images' },
     { role: 'button', name: 'Google apps' },
     { role: 'link', name: 'Sign in' },
     { role: 'combobox',
       name: 'Search',
       description: 'Search',
       focused: true,
       autocomplete: 'both' },
     { role: 'button', name: 'Search by voice' },
     { role: 'button',
       name: 'Google Search',
       description: 'Google Search' },
     { role: 'button',
       name: 'I\'m Feeling Lucky',
       description: 'I\'m Feeling Lucky' },
     { role: 'img', name: 'Google' },
     { role: 'link', name: 'Privacy' },
     { role: 'link', name: 'Terms' },
     { role: 'combobox', name: 'Settings', haspopup: 'menu' },
     { role: 'link', name: 'Advertising' },
     { role: 'link', name: 'Business' } ] }

But more useful would be:

{ role: 'WebArea',
  name: 'Google',
  nodeId: 1,
  children:
   [ { role: 'button', name: 'Dismiss', nodeId: 4 },
     { role: 'link', name: 'About', nodeId: 6 },
     { role: 'link', name: 'Store', nodeId: 7 },
     { role: 'link', name: 'Gmail', nodeId: 8 },
     { role: 'link', name: 'Images', nodeId: 11 },
     { role: 'button', name: 'Google apps', nodeId: 47 },
     { role: 'link', name: 'Sign in', nodeId: 48 },
 // ...
     { role: 'link', name: 'Business', nodeId: 129 } ] }

If approved, I feel I could make the needed changes to submit a PR.

[dtinth]

I need to interact (click, type, etc.) via the accessibility tree, so I need some kind of workaround… Sharing it here:

Workaround as of Puppeteer v5.0.0

Getting a raw accessibility tree from a Puppeteer page

const client = getClient(page)
const tree = await getAccessibilityTree(client)

/**
 * @param {import("puppeteer").Page} page
 */
function getClient(page) {
  return /** @type {import('puppeteer').CDPSession} */ (
    /** @type {any} */ (page)._client
  )
}

/**
 * @param {import("puppeteer").CDPSession} client
 */
async function getAccessibilityTree(client) {
  return /** @type {import("puppeteer/lib/esm/protocol").default.Accessibility.getFullAXTreeReturnValue} */ (await client.send(
    'Accessibility.getFullAXTree',
  ))
}

The result will contain nodes[*].backendDOMNodeId which allows resolving back to a DOM node.

Getting a DOM node given a backendDOMNodeId

const node = await resolveNodeFromBackendNodeId(page.mainFrame(), backendDOMNodeId)

/**
 * @param {import("puppeteer").Frame} frame
 * @param {number} backendNodeId
 */
async function resolveNodeFromBackendNodeId(frame, backendNodeId) {
  const ctx = await Promise.resolve(frame.executionContext())
  return /** @type {import('puppeteer').ElementHandle} */ (
    /** @type {any} */ (ctx)._adoptBackendNodeId(backendNodeId)
  )
}

The result will be an element handle that we can interact with.

[FremyCompany]

@JoelEinbinder I would consider doing that if that was required. Fwiw, your proposed elementHandle.accessibilitySnapshot would work really well for me (I would then navigate the dom as primary, then get the accessibility info on end nodes like input/video where the dom tree isn’t providing the relevant info). If you do that, elementHandle.accessiblitySnapshot should maybe have an optional parameter specifying not to return children nodes as an optimization, just the AXNode associated to the element.