[🐛 Bug]: Grid Protocol Converter Insufficient

What happened?

Java Bindings and the Grid are maintaining a lot of extra code in order to be backwards compatible with earlier versions of Selenium. Except we don’t have any tests to show we’ve been successful with this. I ran the old Watir-WebDriver tests using the final version of Se2 for the Ruby bindings against the latest Grid, and 40% of them failed.

Updated Update

Found several more issues in Java bindings for OSS code; there are 3 categories:

1. Stuff that was broken in Grid 4 with the Locator Conversions (name, id & class name locators)
2. Stuff that was broken in Server 3.7 when translating JWP/early W3C to W3C (switchToWindow & sendKeysToActiveElement) — broken running Java bindings 2.53.1
3. Stuff that was broken in Server 3.7 when translating pre-W3C JWP to W3C (locators except id & name, implicitlyWait, setScriptTimeout and switchToFrame) — broken running Java bindings 2.45.0 or other bindings 2.53.1

This is now looking like a lot more work to fix; and less likely that people are making as much use of the protocol translations as we’ve thought.

Update

• ~The two things definitely affecting the Java bindings actually look like they might be simple to fix~
• I’ve never seen a framework that didn’t have a way to check if an element wasn’t there, and didn’t use id/name/class name to do so, which indicates to me that no one has tried to run their 2.x code on a 4.x server, otherwise they’d have raised an issue.
• No one using Selenium 2 could use Selenium 3 server and switch windows since v3.6 (August 2017), which small sample size, but I’ve never worked anywhere that I didn’t need to switch windows for my test suites.
• I was surprised that Selenium Server 2.53 was able to run latest Chrome version so well. If all you care about is Chrome and/or anything working, there’s zero reason to upgrade off of Server 2.53 right now, (which would explain why there are still so many downloads of it).
• Whatever we end up doing, I don’t think it is unreasonable that running tests on the latest versions of (non-Chrome) browsers should require updating to a version of Selenium > 3.9 when w3c was pretty much finalized, in February 2018.

Uploaded a repo with 13 failing JWP tests with Grid 4 (and 1 failing JWP test with Server v3 for completeness) https://github.com/titusfortner/grid4conversion/tree/main/src/test/java/com/titusfortner/grid4conversion

Options

1. File these as bugs and fix/ignore as time allows until we do a non-backward compatible rewrite in Selenium 5
2. Fix these bugs ASAP and then separate out all the protocol conversion code and provide it the same way we do with leg-rc package
3. Consider it a sunk cost, and just delete all of it now so we can simplify the code and make forward progress

Someone suggested to me that we should proceed with option 3 and if any user really wants option 2, they can pay us/someone to do it, and that makes sense to me with limited resources.

Supporting Data

Here are the different combinations I tested with old watir-webdriver specs using Selenium 2.53.4 Ruby bindings with the latest versions of the browsers and the latest versions Selenium (toggling OSS mode on/off in Se 3/4 for Chrome)

Browser	Local	Se 2	Se 3	Se 3 OSS	Se 4	Se 4 OSS	Total Tests
Chrome	3	1	356	3	489	117	1235
Firefox	N/A	N/A	364	N/A	516	N/A	1244
Safari	N/A	N/A	302	N/A	404	N/A	1120

Things that are broken when running Bindings 2.53.x

• Return JavaScriptEnabled as part of capabilities (Ruby only — won’t even try to execute JS if not in caps)
• Ensure Window “name” parameter is updated to “handle” (Java & Ruby)
• Delete null values from Cookie arguments (Ruby only — Java deletes null params before sending)
• ~Translate click actions with bodies into correct actions (e.g., {"button": 2})~ (This was broken for Selenium Server 3.x but fixed in Grid 4)
• Translate moveTo commands with x/y offsets (Ruby only — Ruby sends null element for just moveTo, Java omits it)
• Using id or name to locate a missing element will throw the wrong error (Java & Ruby)
• Using class name to locate an element gives unable to determine locator strategy (Java & Ruby)
• Sending Key to Active Element gives UnsupportedCommandException (Java & Ruby)

Things that are broken when running Java Bindings 2.45 (or later for other bindings):

• Using css selector, xpath, tag name, link text, or partial link text values can’t be converted to Web Element
• Setting script timeout –> Cannot call non W3C standard command while in W3C mode
• Setting implicit wait timeout —> Cannot call non W3C standard command while in W3C mode
• Switching to Frame by Name —> invalid argument: ‘id’ can not be string

(Watir doesn’t do a whole lot with actions, so it’s ~likely~ possible ~a bunch of~ more things are broken in the translation)

@asolntsev is going to try to help us out by running the same experiment with old Selenide test suite for Java. That might tell us more about how bad the situation is. It’s possible that other bindings will have fewer issues.

My guess though is that our assumption that Selenium 2 Bindings users can use Selenium 3 Server or Selenium 4 Grid is misguided, and that people still using Selenium 2 are using Selenium 2 server, and we would be better off removing the translation code rather than fixing it.

How can we reproduce the issue?

Ruby can be reproduced by running watir-webdriver 0.9.3 watirspecs with selenium-webdriver 2.53.4 setting REMOTE, and updating the server start code. Java with JUnit 5:

    @Test
    void getWindow() {
        Set<String> windowHandles = driver.getWindowHandles();
        String handle = windowHandles.stream().findFirst().get();

        Assertions.assertDoesNotThrow(() -> driver.switchTo().window(handle));
    }

    @Test
    void useIDForMissingElement() {
        NoSuchElementException thrown = Assertions.assertThrows(NoSuchElementException.class, () -> {
            driver.findElement(By.id("not-there"));
        });

        String msg = "no such element: Unable to locate element: {\"method\":\"id\",\"selector\":\"not-there\"}";
        Assertions.assertTrue(thrown.getMessage().contains(msg));
    }

Relevant log output

Caused by: org.openqa.selenium.WebDriverException: Returned value cannot be converted to WebElement: {stacktrace=java.lang.RuntimeException: Unable to execute request for an existing session: An unknown error has occurred

and

org.opentest4j.AssertionFailedError: Unexpected exception thrown: org.openqa.selenium.WebDriverException: invalid argument: 'handle' must be a string

Operating System

MacOS Big Sur

Selenium version

Ruby 2.53.4 & Java 2.45.0 & Java 2.53.1

What are the browser(s) and version(s) where you see this issue?

Chrome, Firefox, Safari

What are the browser driver(s) and version(s) where you see this issue?

Latest

Are you using Selenium Grid?

4.1.2

[titusfortner]

Ah, yes, for those, you’ll need to construct your own MutableCapabilities instance. It’s likely we’ll eventually deprecate DesiredCapabilities from RemoteWebDriver like we did local driver classes, but will continue to support MutableCapabilities.

Please find us on Slack/IRC to discuss more: https://www.selenium.dev/support/#ChatRoom

[titusfortner]

It will still work. The deprecation notices are much more eager than we realized when we released.