Testem Chrome Timeouts

Hey,

as I mention in https://github.com/testem/testem/issues/1015 we frequently get testem failures in CI when running against the Chrome (we use Testem through Ember CLI). Our test runs either fail to start (as if testem can’t find the browser) or fail to finish (all tests have run and pass, but the browser window sits there and it eventually times out). It usually just tells us testem.js has not loaded.

As the issue is intermittent it can be hard to reproduce. I’ve tried to recreate it in a repo by simply forking Travis CI’s web repo (uses Ember CLI) and adding ember-exam to it (Ember Exam just allows you to easily use Testem’s parallel feature). If running a single test run it’d probably take some time to get a timeout, but if you run ember exam --split=9 --parallel you should start getting some timeouts in a few runs.

I’ve generated some (pretty verbose) logs here.

Generally on startup I’ve seen the following errors:

not ok 323 Chrome - error
    ---
        message: >
            Error: Browser failed to connect within 30s. testem.js not loaded?
            Stderr:
             2016-11-19 00:42:01.877 Google Chrome[28575:622538] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x7fa45b8022a0>. Break on NSLog to debug.
            2016-11-19 00:42:01.878 Google Chrome[28575:622538] Call stack:
            (
                "+callStackSymbols disabled for performance reasons"
            )


        Log: |
            { type: 'error',
              text: 'Error: Browser failed to connect within 30s. testem.js not loaded?' }
            { type: 'error',
              text: '2016-11-19 00:42:01.877 Google Chrome[28575:622538] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x7fa45b8022a0>. Break on NSLog to debug.\n2016-11-19 00:42:01.878 Google Chrome[28575:622538] Call stack:\n(\n    "+callStackSymbols disabled for performance reasons"\n)\n' }
    ...

On timeouts at the end I’ve seen

  engine:ws writing "3" +1ms
  engine:polling closing +649ms
  engine:polling transport not writable - buffering orderly close +0ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
  engine:polling closing +8ms
  engine:polling transport not writable - buffering orderly close +1ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
  engine:polling closing +654ms
  engine:polling transport not writable - buffering orderly close +0ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
  engine:polling closing +78ms
  engine:polling transport not writable - buffering orderly close +0ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
  engine:polling closing +5s
  engine:polling transport not writable - buffering orderly close +0ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
  engine:polling closing +2s
  engine:polling transport not writable - buffering orderly close +0ms
  socket.io:client client close with reason ping timeout +0ms
  socket.io:socket closing socket - reason ping timeout +0ms
not ok 205 Chrome - error
    ---
        message: >
            Error: Browser disconnected
            Stderr:
             2016-11-19 00:49:10.060 Google Chrome[28965:629220] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x7fa7a07af8a0>. Break on NSLog to debug.
            2016-11-19 00:49:10.061 Google Chrome[28965:629220] Call stack:
            (
                "+callStackSymbols disabled for performance reasons"
            )


        Log: |
            { type: 'error', text: 'Error: Browser disconnected' }
            { type: 'error',
              text: '2016-11-19 00:49:10.060 Google Chrome[28965:629220] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x7fa7a07af8a0>. Break on NSLog to debug.\n2016-11-19 00:49:10.061 Google Chrome[28965:629220] Call stack:\n(\n    "+callStackSymbols disabled for performance reasons"\n)\n' }
    ...
  socket.io:client client close with reason transport close +8s
  socket.io:socket closing socket - reason transport close +0ms

Other things to note:

• we’ve been experiencing this for a while now, through multiple versions of Chrome, Node and Testem. Currently running latest Chrome, Testem 1.13 and Node 7.
• We have a large codebase (~150K LOC) so not sure if that amplifies the issue or not.

/cc @stefanpenner (as we chatted briefly about this)

[bricss]

This is how I fixed timeout issue with Chrome 59+:

{
    "framework": "qunit",
    "test_page": "tests/index.html?hidepassed",
    "disable_watching": true,
    "launch_in_ci": [
        "Chrome"
    ],
    "launch_in_dev": [
        "Chrome"
    ],
    "browser_start_timeout": 60,
    "browser_args": {
        "Chrome": [
            "--disable-gpu",
            "--disable-web-security", // optional, since chrome 60+
            "--headless",
            "--incognito", // optional
            "--no-sandbox", // optional, since chrome 60+
            "--remote-debugging-address=0.0.0.0",
            "--remote-debugging-port=9222"
        ]
    }
}

[mani-mishra]

We are using ember exam and also seeing similar issues with our Team City build server. The error message is slightly different though.

[Chrome - error] Error: Browser failed to connect within 30s. testem.js not loaded?
Stderr: 
 [0419/061938.039012:WARNING:audio_manager.cc(321)] Multiple instances of AudioManager detected
[0419/061938.039065:WARNING:audio_manager.cc(278)] Multiple instances of AudioManager detected
[0419/061938.371223:ERROR:socket_posix.cc(126)] bind() returned an error, errno=98: Address already in use
[0419/061938.371271:ERROR:devtools_http_handler.cc(221)] Cannot start http server for devtools. Stop devtools.