UIAutomator2 crashes sometimes, causing "socket hang up"

The problem

We are running an inhouse device farm, and encounter “socket hang up” errors from time to time.

Everyday we run about 800 test cases spreadly on 10 devices. Sometimes we got only some cases failed with this error, sometimes we failed ~20%, unpredictably.

Some devices are tough, only get this error once or twice. Some others are more fragile and get this error every now and then.

We tried to dive deeply into Appium’log and found this:

[HTTP] --> POST /wd/hub/session/f1daf4c7-4ef4-45df-ab0d-ebe6722e1f80/element
[HTTP] {"using":"id","value":"notification_container"}
[W3C (f1daf4c7)] Calling AppiumDriver.findElement() with args: ["id","notification_container","f1daf4c7-4ef4-45df-ab0d-ebe6722e1f80"]
[BaseDriver] Valid locator strategies for this request: xpath, id, class name, accessibility id, -android uiautomator
[BaseDriver] Waiting up to 0 ms for condition
[WD Proxy] Matched '/element' to command name 'findElement'
[WD Proxy] Proxying [POST /element] to [POST http://127.0.0.1:8201/wd/hub/session/953fff26-affe-4ffa-9284-11ef8124c9f5/element] with body: {"strategy":"id","selector":"notification_container","context":"","multiple":false}
[Instrumentation] INSTRUMENTATION_RESULT: shortMsg=Process crashed.
[Instrumentation] INSTRUMENTATION_CODE: 0
[WD Proxy] socket hang up
[W3C (f1daf4c7)] Encountered internal error running command: UnknownError: An unknown server-side error occurred while processing the command. Original error: Could not proxy command to the remote server. Original error: socket hang up
[W3C (f1daf4c7)]     at JWProxy.command (/home/linuxbrew/.linuxbrew/lib/node_modules/appium/node_modules/appium-base-driver/lib/jsonwp-proxy/proxy.js:280:13)
[W3C (f1daf4c7)]     at runMicrotasks (<anonymous>)
[W3C (f1daf4c7)]     at processTicksAndRejections (internal/process/task_queues.js:97:5)
[W3C (f1daf4c7)]     at AndroidUiautomator2Driver.helpers.doFindElementOrEls (/home/linuxbrew/.linuxbrew/lib/node_modules/appium/node_modules/appium-uiautomator2-driver/lib/commands/find.js:27:12)
[W3C (f1daf4c7)]     at doFind (/home/linuxbrew/.linuxbrew/lib/node_modules/appium/node_modules/appium-android-driver/lib/commands/find.js:36:17)
[W3C (f1daf4c7)]     at wrappedCondFn (/home/linuxbrew/.linuxbrew/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/timeout.js:151:12)
[W3C (f1daf4c7)]     at spin (/home/linuxbrew/.linuxbrew/lib/node_modules/appium/node_modules/asyncbox/lib/asyncbox.js:113:20)
[W3C (f1daf4c7)]     at waitForCondition (/home/linuxbrew/.linuxbrew/lib/node_modules/appium/node_modules/asyncbox/lib/asyncbox.js:129:10)
[W3C (f1daf4c7)]     at AndroidUiautomator2Driver.implicitWaitForCondition (/home/linuxbrew/.linuxbrew/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/timeout.js:153:10)
[W3C (f1daf4c7)]     at AndroidUiautomator2Driver.findElOrEls (/home/linuxbrew/.linuxbrew/lib/node_modules/appium/node_modules/appium-android-driver/lib/commands/find.js:60:5)
[W3C (f1daf4c7)]     at AndroidUiautomator2Driver.findElOrElsWithProcessing (/home/linuxbrew/.linuxbrew/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/find.js:33:12)
[W3C (f1daf4c7)]     at AndroidUiautomator2Driver.findElement (/home/linuxbrew/.linuxbrew/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/find.js:53:10)
[HTTP] <-- POST /wd/hub/session/f1daf4c7-4ef4-45df-ab0d-ebe6722e1f80/element 500 162 ms - 2041

This seems to be similar to issue https://github.com/appium/appium/issues/10749, and seems to be fixed in https://github.com/appium/appium-uiautomator2-driver/blob/80216fce6c9850ef14ee27284de4ab5cb6094ac6/lib/uiautomator2.js#L321.

But it still happens.

Environment

• Appium version (or git revision) that exhibits the issue: 1.18.3
• Desktop OS/version used to run Appium: Ubuntu 18.04/20.04
• Node.js version (unless using Appium.app|exe): 15.2.1
• Npm or Yarn package manager: Npm
• Mobile platform/version under test: Android 9/10
• Real device or emulator/simulator: Read devices
• Appium CLI or Appium.app|exe: Appium CLI

Details

If necessary, describe the problem you have been experiencing in more detail.

Link to Appium logs

https://gist.github.com/ltpquang/ce8689e433529aea585c602682deba14

[mykola-mokhnach]

there might be multiple reasons for the instrumentation crash. Consider checking the logcat output to find possible traces there.

[ltpquang]

After opening this issue, I tried solution suggested by @dineshchakravarthy5 in https://github.com/appium/appium/issues/10749: explicitly wait for 10 seconds before creating a new session. Moreover, I also tried to start a new Appium server instance everytime the pipelines start to run, instead of having one instance running 24/7. Since then, nearly 100 test runs with ~5000 sessions created with only a few “socket hang up” occurrences. Maybe you folks can tried and comment back so the latter can get some hints to resolve this mystery.