Selecting elements for screenshotting based on tag content

Selecting elements as the target for screenshots based on CSS selectors does not (currently) allow for the selection of elements based on tag content, or relative to DOM elements selected based on tag content.

However, elements can be selected based on tag content using Javascript. It would be useful to allow for the selection of elements via Javascript as well as CSS.

Alternatively, support a method that can be called from a javascript scraper call that will apply a screen shot to a Javascript selected element.

At the moment, selector based screenshots seem to be focused in _selector_javascript(selectors) by:

https://github.com/simonw/shot-scraper/blob/e40070d9bc5334ad8b2ff9e4e3565263aad545f4/shot_scraper/cli.py#L520

As well as passing --selector(s), s, one approach might be to pass element(s) el returned from a --js-selector script?

[simonw]

OK, I built a protoype. @psychemedia you can try it out by running:

pip install https://github.com/simonw/shot-scraper/archive/8a61c7e1ba90f1210868cf7f84ece78e5f8300db.zip

Here’s a demo:

shot-scraper https://simonwillison.net/2022/Apr/8/weeknotes/ \
  --js-selector 'el.tagName == "P" && el.innerText.includes("shot-scraper")' \
  --padding 15 --retina

Which produced this image (of the first paragraph on the page to contain the term shot-scraper):

Note that I had to include el.tagName == "P" in the expression - because it scans through every tag on the page, so without that it returns the html tag since that includes the shot-scraper text somewhere in its innerText!

The demo also currently outputs debug info showing the JavaScript it generated - I’d remove that before shipping the feature:

% shot-scraper https://simonwillison.net/2022/Apr/8/weeknotes/ \
      --js-selector 'el.tagName == "P" && el.innerText.includes("shot-scraper")' \
      --padding 15 --retina
() => {
Array.from(
  document.getElementsByTagName('*')
).find(el => el.tagName == "P" && el.innerText.includes("shot-scraper")).classList.add("js-selector-a1f5ba0fc4a4317e58a3bd11a0f16b96");
} ['.js-selector-a1f5ba0fc4a4317e58a3bd11a0f16b96']
Screenshot of '.js-selector-a1f5ba0fc4a4317e58a3bd11a0f16b96' on 'https://simonwillison.net/2022/Apr/8/weeknotes/' written to 'simonwillison-net-2022-Apr-8-weeknotes.1.png'

[simonw]

Documentation: https://github.com/simonw/shot-scraper/blob/59b156ddd7963fd19edca214edc0ac59552efbb2/README.md#specifying-elements-using-javascript-filters