Requesting multiple sessions on the same device on a single Appium server breaks parallel execution in combination with Selenium Grid

The problem

When requesting a session multiple times for the same device on an Appium server, the behavior is reproducibly faulty. After requesting a session for a specific device (with capabilities such as UDID and platformVersion), a second request to the same device overrides the previous test instance. This behavior remains unchanged regardless of the value of the --session-override argument (explicitly tried with both ‘true’ and ‘false’, even though the default value which we require for our test service is ‘false’ anyway). maxInstances is also set to 1.

This behavior becomes especially problematic in combination with a Selenium Grid. After executing the procedure described above with the Appium server running on a node of the Selenium Grid, the Grid will show two sessions as being active, even though that’s not possible due to one device only allowing one test instance.

Parallel execution on multiple devices on the same node/server are possible according to official documentation. However each time an issue with this approach is brought up, the response is to instead just use one Appium server per device, which would come with a lot of overhead in our case.

Our guess so far is that, in addition to the above mentioned override behavior, there is a miscommunication between the Selenium Grid and the Appium server.

Environment

• Appium version: Appium v2.0.0-beta.25
• Last Appium version that did not exhibit the issue: All Appium versions (such as v1.22.0) we have tried exhibited issues with parallel exception on the same node.
• Desktop OS/version used to run Appium: Tried MAC and Windows
• Node.js version: v16.13.2
• Npm: npm 8.1.2
• Mobile platform/version under test: Android 10/Android 11
• Real device or emulator/simulator: Tried with a combination of emulated and real Pixel and Samsung Galaxy devices respectively.
• Appium CLI or Appium.app|exe: CLI, app and exe

Details

Setup

• We have a physical Android device, and an emulated Android device connected to our system. adb devices -l delivers the following output:

List of devices attached
RZ8NB135G4Y device product:r8seea model:SM_G780F device:r8s transport_id:1
emulator-5554 device product:sdk_gphone_x86 model:Android_SDK_built_for_x86 device:generic_x86 transport_id:2

• In the case of using a Selenium Grid, we are starting the Selenium Hub and Appium Relay within a docker container with the following docker-compose:

version: "3.7"

services:
  selenium-hub:
    image: selenium/hub:4
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"
    healthcheck:
      test: /opt/bin/check-grid.sh
      interval: 30s
      timeout: 30s
      retries: 5
      start_period: 60s
    environment:
      - SE_NODE_SESSION_TIMEOUT=300
      - SE_SESSION_REQUEST_TIMEOUT=30
      - SE_SESSION_RETRY_INTERVAL=5


  appium-relay:
    image: selenium/node-docker:4
    container_name: appium-relay
    volumes:
      - C:\Users\xxx\Downloads\swissium\config.toml:/opt/bin/config.toml
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    extra_hosts:
      - "host.docker.internal:host-gateway"

• The Appium Relay allows two sessions and a maximum of 1 instances per device. See the following additional config file:

[node]
detect-drivers = false
max-sessions = 2

[relay]
url = "http://host.docker.internal:4723"
status-endpoint = "/status"
configs = [
	'1', '{"browserName": "samsung SM-G780F", "platformName": "Android", "appium:platformVersion": "10", "udid": "RZ8NB135G4Y"}',
	'1', '{"browserName": "Google Android SDK built for x86", "platformName": "Android", "appium:platformVersion": "10", "udid": "emulator-5554"}'
]

• We are testing the behavior separately from our code, by simply issuing requests to the Selenium Grid/Appium Server via Postman.

Direct Communication with Appium server:

To verify that there’s no issue with our setup, we tried requesting parallel sessions by communicating directly with the Appium server (no Selenium Grid involved). After having the Appium server running with the devices described above, following steps are taken:

1. Send a request, specifically requesting the emulated Pixel device through UDID and Android version capabilities. Output: We receive a session on the requested device. (Details)

2. Send a second request, specifically requesting the emulated Pixel device through UDID and Android version capabilities again, merely changing the appium:systemPort capability. Output: We receive a session on the same device(Details), despite session-override being disabled.

Observed behavior: After the first request, a browser window opens on the emulated Pixel device. Upon second request, the browser windows closes and opens up again on the same device, while the Samsung device exhibits no behavior.

Appium Log

Communication through Selenium Grid:

1. Start the Selenium Grid with the above mentioned setup. We can see the following expected configuration of our Selenium Grid:

2. Send a request, specifically requesting the physical Samsung device through UDID and Android Version capabilities. Output: We receive a session on the requested device(Details). Our Selenium Grid shows one session on the correct device.

3. Send a second request, specifically requesting the physical Samsung device through UDID and Android Version capabilities again, merely changing the appium:systemPort capability. Output: We receive a session on the same device, despite session-override being disabled (Details). Now our Selenium Grid shows TWO sessions in use (both showing the UDID of the Samsung device in their capabilities).

Observed behavior: After the first request, a browser window opens on the physical Samsung device. Upon second request, the browser windows closes and opens up again on the same device, while the Pixel device exhibits no behavior.

Appium Log Selenium Hub Log Selenium Relay Node Log

Needless to say our grid is now at full capacity, it won’t be able to start another session on the Pixel device now, despite it being impossible for two sessions to run on the same device, especially considering we set maxInstances=1. This behavior seems unintended and also breaks parallel execution when running multiple devices on one Appium server as a node of a Selenium Grid.

We have tried the above procedures with multiple combinations of OS, physical and simulated devices, as well as different Selenium and Appium versions. It is noteworthy that the behavior with older Selenium and Appium versions was far more unexpected and bizarre than the behavior documented here, however for the sake of keeping this ticket at a reasonable length we won’t be including it.

Link to Appium logs

• Appium Log for direct communication with Appium server

• Appium Log for communication through Selenium Grid

[diemol]

Without having a deeper look, what I can see in the posted config.toml is that "udid": "RZ8NB135G4Y" is not prefixed with appium:. Therefore, it won’t be used for matching because it is not a compliant W3C capability.

What happens when you prefix it?

[Trigtrig]

The prefix in the config worked. Updated Config:

detect-drivers = false
max-sessions = 2

[relay]
url = "http://host.docker.internal:4723"
status-endpoint = "/status"
configs = [
	'1', '{"browserName": "samsung SM-G780F", "platformName": "Android", "appium:platformVersion": "10", "appium:udid": "RZ8NB135G4Y"}',
	'1', '{"browserName": "Google Android SDK built for x86", "platformName": "Android", "appium:platformVersion": "10", "appium:udid": "emulator-5554"}'
]

Now the first session request is successful while the second request fails as expected. Here is the excerpt of the second request from Postman:

POST /session HTTP/1.1
Content-Type: application/json
User-Agent: PostmanRuntime/7.29.0
Accept: */*
Cache-Control: no-cache
Postman-Token: 122dd1ff-846d-4708-8d97-6ddc40e777d4
Host: localhost:4444
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 595
 
{
"capabilities": {
"alwaysMatch": {
"appium:appActivity": "com.google.android.apps.chrome.Main",
"appium:appPackage": "com.android.chrome",
"appium:automationName": "UIAutomator2",
"appium:ensureWebviewsHavePages": true,
"appium:language": "de",
"appium:locale": "DE",
"appium:newCommandTimeout": 300,
"appium:platformVersion": "10",
"platformName": "android",
"appium:udid": "RZ8NB135G4Y",
"appium:systemPort": 8102
}
}
}
 
HTTP/1.1 500 Internal Server Error
content-length: 9442
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
 
{
"value": {
"error": "session not created",
"message": "Could not start a new session. New session request timed out\nBuild info: version: '4.1.2', revision: '9a5a329c5a'\nSystem info: host: '28864e245c72', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '5.10.16.3-microsoft-standard-WSL2', java.version: '11.0.13'\nDriver info: driver.version: unknown",
"stacktrace": [
{
"fileName": "LocalNewSessionQueue.java",
"moduleVersion": null,
"moduleName": null,
"nativeMethod": false,
"methodName": "addToQueue",
"className": "org.openqa.selenium.grid.sessionqueue.local.LocalNewSessionQueue",
"lineNumber": 200,
"classLoaderName": "app"
},

... 

Thanks a lot for the help