[IOS, XCUnitTest] has anyone else noticed degrading performance in repeatedly launching test script?

The problem

Following on from https://github.com/appium/appium/issues/7192 being reported yesterday as also happening on 64 bit devices, I decided to writ a very simple looper to see if i can determine how many restarts we get before a physical reboot is needed.

The test case is as follows (simplified from real script so may not be syntactically correct):

caps= {'deviceName': device_name, 'platformName': 'iOS', 'newCommandTimeout': 3600, 'udid': device_id, 'fullReset': False, 'automationName': 'xcuitest', 'platformVersion': "10.2.1", 'realDeviceLogger': 'idevicesyslog', 'bundleId': 'com.google.ios.youtube'}
for loop in testloops:
    loopstart = datetime.datetime.now()
    launchstart = datetime.datetime.now()
    driver = webdriver.Remote("http://localhost:4723/wd/hub", caps)
    launchend = datetime.datetime.now()
    launchtime = (launchend-launchstart).seconds
    logger.info("Created webdriver remote in {0} seconds".format(launchtime))
    driver.save_screenshot("launch{0:04}_screenshot.png".format(loop))
    sleep(5)
    driver.close_app()
    loopend = datetime.datetime.now()
    looptime = (loopend - loopstart).seconds
    logger.debug("END OF LOOP : {0} seconds duration".format(looptime))

Environment

• Appium version (or git revision) that exhibits the issue: 1.6.4-beta
• Last Appium version that did not exhibit the issue (if applicable): unknown
• Desktop OS/version used to run Appium: OSX 10.11.6
• Node.js version (unless using Appium.app|exe): 6.5.0
• Mobile platform/version under test: IOS 10.2.1
• Real device or emulator/simulator: Real
• Appium CLI or Appium.app|exe: appium python bindings

I’ve set this up to perform 2000 test loops. Currently on loop 31 of these (had previously got to 65 and noticed the degrading performance so restarted with the timings printed out) and these are the results so far (I’ve abbreviated after the middle ones to make it shorter…):

2017-03-23 11:01:58,824 - DEBUG - ///////////////// Test run 0 of 2000 \\\\\\\\\\
2017-03-23 11:03:57,667 - INFO - Created webdriver remote in 118 seconds
2017-03-23 11:04:03,818 - DEBUG - ****** END OF LOOP : 124 seconds duration ******

Loop:1; created remote: 15 seconds; loop duration: 21 seconds 
Loop:2; created remote: 17 seconds; loop duration: 22 seconds 
Loop:3; created remote: 16 seconds; loop duration: 23 seconds 
Loop:4; created remote: 17 seconds; loop duration: 24 seconds 
Loop:5; created remote: 19 seconds; loop duration: 26 seconds 
Loop:6; created remote: 20 seconds; loop duration: 28 seconds 
Loop:7; created remote: 19 seconds; loop duration: 27 seconds 
Loop:8; created remote: 21 seconds; loop duration: 29 seconds 
Loop:9; created remote: 19 seconds; loop duration: 28 seconds 
Loop:10; created remote: 26 seconds; loop duration: 35 seconds 
Loop:11; created remote: 21 seconds; loop duration: 30 seconds 
Loop:12; created remote: 26 seconds; loop duration: 36 seconds 
Loop:13; created remote: 23 seconds; loop duration: 33 seconds 
Loop:14; created remote: 25 seconds; loop duration: 36 seconds 
Loop:15; created remote: 32 seconds; loop duration: 43 seconds 
Loop:16; created remote: 28 seconds; loop duration: 40 seconds 
Loop:17; created remote: 36 seconds; loop duration: 49 seconds 
Loop:18; created remote: 39 seconds; loop duration: 54 seconds 
Loop:19; created remote: 45 seconds; loop duration: 56 seconds 
Loop:20; created remote: 53 seconds; loop duration: 69 seconds 
Loop:21; created remote: 47 seconds; loop duration: 63 seconds 
Loop:22; created remote: 49 seconds; loop duration: 66 seconds 
Loop:23; created remote: 67 seconds; loop duration: 84 seconds 
Loop:24; created remote: 47 seconds; loop duration: 59 seconds 
Loop:25; created remote: 45 seconds; loop duration: 62 seconds 
Loop:26; created remote: 59 seconds; loop duration: 78 seconds 
Loop:27; created remote: 60 seconds; loop duration: 78 seconds 
Loop:28; created remote: 66 seconds; loop duration: 84 seconds 
Loop:29; created remote: 57 seconds; loop duration: 77 seconds 
Loop:30; created remote: 69 seconds; loop duration: 89 seconds

2017-03-23 11:28:28,748 - DEBUG - ///////////////// Test run 31 of 2000 \\\\\\\\\\
2017-03-23 11:29:36,779 - INFO - Created webdriver remote in 68 seconds
2017-03-23 11:29:56,908 - DEBUG - ****** END OF LOOP : 88 seconds duration ******

and in fact since i started writing this it’s up to loop 40:

017-03-23 11:41:25,672 - test - DEBUG - ///////////////// Test run 40 of 2000 \\\\\\\\\\
2017-03-23 11:42:38,849 - test - INFO - Created webdriver remote in 73 seconds
2017-03-23 11:43:01,001 - test - DEBUG - ****** END OF LOOP : 95 seconds duration ******

So as you can see, it’s quite quickly degrading in both the launch time of the application, and I presume the time to take the screenshot is also increasing as there’s nothing else happening in the loop… ( I consider the first one an outlier as I had just booted the ipad, so maybe it wasn’t fully ready)

Has anyone else noticed similar behaviour?

[Brian-Watson]

@kenmcc So are you launching an appium server process and then running 100s of tests against it?

I’ve found that even with the older versions of Appium, after running for a while, performance goes down and flakey errors start popping up. I’ve actually re-worked my automation to start a new Appium process for each test suite I run. Each suite has 1 to 10 tests in it. Once the suite is over, I kill the process so the next suite gets a new one. Since I’ve started doing this, my runs are much more stable. It take a little longer to execute since you have the extra time it takes to start and stop the Appium process, but it’s ultimately worth it in the end.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.