[🐛 Bug]: driver.execute_script() does not work with chromedriver/msedgedriver v96 - return dict and not WebDriver object

What happened?

After updating chrome and/or msedge to v96, driver.execute_script() no longer works it returns a dict(hash) instead of webDriver element:

I need to execute a shadowRoot pseudo-element in order to get the properties inside the element.

I think there are changes in the browser driver that prevent this from working, I think this can be reproduceable with any pseudo-elements that needs javascript executing.

I’ve tried all combinations of:

options.w3c = True|False
options.add_experimental_option('w3c', True|False)

Current workaround is not updating beyond v95 of chrome/msedge

How can we reproduce the issue?

Example code python link to one of our customer test logins, trying to click a button behins a jsdom.

#!/usr/bin/env python3
import sys
import selenium
import selenium.webdriver
from selenium.webdriver.common.by import By

url = "https://www.trk.qa-04.buypass.no/key-registration"

# options = selenium.webdriver.EdgeOptions()
# options.binary_location = "C:\\Program Files (x86)\\Microsoft\\Edge Dev\\Application\\msedge.exe"
# #options.w3c = False
# #options.add_experimental_option('w3c', True)
# options.use_chromium = True
# wd = selenium.webdriver.Edge(options=options, service_args=['--verbose'])


# default chrome
options = selenium.webdriver.ChromeOptions()
options.binary_location = "C:\\Program Files (x86)\\Google\\Chrome Beta\\Application\\chrome.exe"
#options.w3c = False
#options.add_experimental_option('w3c', True)
wd = selenium.webdriver.Chrome(options=options)

wd.set_window_size(800,800)
print(f"webdriver:{wd.name}, capabilities:{wd.capabilities}")
print(f"type:{type(wd)}")
print(f"python version:{sys.version}")
print(f"selenium version:{selenium.__version__}")

wd.get(url)
el = wd.find_element(By.XPATH, "//bp-button[@text='Buypass ID']")
shadow_root = wd.execute_script(f"return arguments[0].shadowRoot", el)
print(f"shadow_root:{shadow_root}")
shadow_root.find_element(By.CSS_SELECTOR, "button").click()

Relevant log output

chrome v95 works as all previous versions:

python .\bin\chrome-selenium-execute-bug.py

DevTools listening on ws://127.0.0.1:54262/devtools/browser/7904da2f-0317-49fb-a07d-2e7706c20324
[15352:11568:1109/133205.386:ERROR:display_layout.cc(559)] PlacementList must be sorted by first 8 bits of display_id 
[15352:11568:1109/133205.700:ERROR:display_layout.cc(559)] PlacementList must be sorted by first 8 bits of display_id 
webdriver:chrome, capabilities:{'acceptInsecureCerts': False, 'browserName': 'chrome', 'browserVersion': '95.0.4638.69', 'chrome': {'chromedriverVersion': '95.0.4638.17 (a9d0719444d4b035e284ed1fce73bf6ccd789df2-refs/branch-heads/4638@{#178})', 'userDataDir': 'C:\\Users\\mobj\\AppData\\Local\\Temp\\scoped_dir14728_516592648'}, 'goog:chromeOptions': {'debuggerAddress': 'localhost:54262'}, 'networkConnectionEnabled': False, 'pageLoadStrategy': 'normal', 'platformName': 'windows', 'proxy': {}, 'setWindowRect': True, 'strictFileInteractability': False, 'timeouts': {'implicit': 0, 'pageLoad': 300000, 
'script': 30000}, 'unhandledPromptBehavior': 'dismiss and notify', 'webauthn:extension:credBlob': True, 'webauthn:extension:largeBlob': True, 'webauthn:virtualAuthenticators': True}
type:<class 'selenium.webdriver.chrome.webdriver.WebDriver'>
python version:3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)]
selenium version:4.0.0
shadow_root:<selenium.webdriver.remote.webelement.WebElement (session="42ee4e062c26b95006696a89773fd37e", element="c1b47141-61c3-40bc-88e8-0fa2ea2cde5b")>

chrome v96 does not work:

python .\bin\chrome-selenium-execute-bug.py
DevTools listening on ws://127.0.0.1:55116/devtools/browser/67905cf2-00b9-4455-9744-6ccd48f654a0
[12352:21508:1109/140012.902:ERROR:display_layout.cc(559)] PlacementList must be sorted by first 8 bits of display_id 
[12352:21508:1109/140013.144:ERROR:display_layout.cc(559)] PlacementList must be sorted by first 8 bits of display_id 
webdriver:chrome, capabilities:{'acceptInsecureCerts': False, 'browserName': 'chrome', 'browserVersion': '96.0.4664.35', 'chrome': {'chromedriverVersion': '96.0.4664.35 (8df3babf8da8faa8aacb78f4848ef66a93b4944b-refs/branch-heads/4664@{#680})', 'userDataDir': 'C:\\Users\\mobj\\AppData\\Local\\Temp\\scoped_dir15676_76951379'}, 'goog:chromeOptions': {'debuggerAddress': 'localhost:55116'}, 'networkConnectionEnabled': False, 'pageLoadStrategy': 'normal', 'platformName': 'windows', 'proxy': {}, 'setWindowRect': True, 'strictFileInteractability': False, 'timeouts': {'implicit': 0, 'pageLoad': 300000, 'script': 30000}, 'unhandledPromptBehavior': 'dismiss and notify', 'webauthn:extension:credBlob': True, 'webauthn:extension:largeBlob': True, 'webauthn:virtualAuthenticators': True}
type:<class 'selenium.webdriver.chrome.webdriver.WebDriver'>
python version:3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)]
selenium version:4.0.0
shadow_root:{'shadow-6066-11e4-a52e-4f735466cecf': '07cda0b2-3936-4421-91fa-c0a69d96aa93'}
Traceback (most recent call last):
  File "C:\dist\work\trk-fullstack-test\bin\chrome-selenium-execute-bug.py", line 31, in <module>
    shadow_root.find_element(By.CSS_SELECTOR, "button").click()
AttributeError: 'dict' object has no attribute 'find_element'

msedgev97 does not work:

python .\bin\chrome-selenium-execute-bug.py
[264:11416:1109/134247.963:ERROR:clean_exit_beacon.cc(241)] beacon_file_path_: C:\Users\mobj\AppData\Local\Temp\scoped_dir3512_1790376771\Variations
[264:11416:1109/134247.966:ERROR:clean_exit_beacon.cc(141)] got_beacon_file_contents: 0

DevTools listening on ws://127.0.0.1:54551/devtools/browser/c5b4e18b-57bc-4eb0-be57-5a1dbe9b5438
 [264:11416:1109/134249.808:ERROR:fallback_task_provider.cc(118)] Every renderer should have at least one task provided by a primary task provider. If a fallback task is shown, it is a bug. Please file a new bug and tag it as a dependency of crbug.com/739782.
webdriver:msedge, capabilities:{'acceptInsecureCerts': False, 'browserName': 'msedge', 'browserVersion': '97.0.1060.2', 'ms:edgeOptions': {'debuggerAddress': 'localhost:54551'}, 'msedge': {'msedgedriverVersion': '97.0.1060.2 (0365cba3e5dc1a8b30b892df0cf076eee17919ce)', 'userDataDir': 'C:\\Users\\mobj\\AppData\\Local\\Temp\\scoped_dir3512_1790376771'}, 'networkConnectionEnabled': False, 'pageLoadStrategy': 'normal', 'platformName': 'windows', 'proxy': {}, 'setWindowRect': True, 'strictFileInteractability': False, 'timeouts': {'implicit': 0, 'pageLoad': 300000, 'script': 30000}, 'unhandledPromptBehavior': 'dismiss and notify', 'webauthn:extension:credBlob': True, 'webauthn:extension:largeBlob': True, 'webauthn:virtualAuthenticators': True}
type:<class 'selenium.webdriver.edge.webdriver.WebDriver'>
python version:3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)]
selenium version:4.0.0
[264:11416:1109/134251.810:ERROR:fallback_task_provider.cc(118)] Every renderer should have at least one task provided by a primary task provider. If a fallback task is shown, it is a bug. Please file a new bug and tag it as a dependency of crbug.com/739782.
shadow_root:{'shadow-6066-11e4-a52e-4f735466cecf': 'a6a68841-eb92-420e-9918-347bf9a70814'}
Traceback (most recent call last):
  File "C:\dist\work\trk-fullstack-test\bin\chrome-selenium-execute-bug.py", line 30, in <module>
    shadow_root.find_element(By.CSS_SELECTOR, "button").click()
AttributeError: 'dict' object has no attribute 'find_element'

Operating System

windows10, ubuntu20.04LTS

Selenium version

4.0.0

What are the browser(s) and version(s) where you see this issue?

Chrome96, MSEdge96, MSEdge97

What are the browser driver(s) and version(s) where you see this issue?

chromedriverVersion: '96.0.4664.35, msedgedriverVersion: 97.0.1060.2

Are you using Selenium Grid?

Na

[AutomatedTester]

Fixed in 0c109931349c9884e929e8ea334104456a6f0547 for python.

[phylor]

Can confirm that the problem exists with Chrome v96, but not in Chrome v95.

If you happen to be on a Debian based system, the following might work to downgrade Chrome:

CHROME_VERSION=95.0.4638.69-1
wget --no-verbose -O /tmp/chrome.deb https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_${CHROME_VERSION}_amd64.deb
sudo apt remove -y google-chrome-stable
sudo apt install -y /tmp/chrome.deb