Waiting for Storybook readiness does not work in headless Firefox

Hello! I’m having difficulties running Creevey tests on Selenium with headless Firefox on Linux. Creevey is trying to start firefox and after a minite it fails with timeout

[CreeveyWorker]: Starting firefox:31362
[FAIL:31362] ScriptTimeoutError: Timed out after 60000 ms
    at Object.throwDecodedError (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/error.js:517:15)
    at parseHttpResponse (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/http.js:642:13)
    at Executor.execute (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/http.js:568:28)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at mixin.execute (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/webdriver.js:731:17)
    at runSequence (/Users/nitive/Work/s7_ds/node_modules/creevey/lib/server/utils.js:156:22)
    at getBrowser (/Users/nitive/Work/s7_ds/node_modules/creevey/lib/server/selenium/browser.js:505:5)
    at Object.worker (/Users/nitive/Work/s7_ds/node_modules/creevey/lib/server/worker/worker.js:196:19)

I started to debug what causes the timeout and found out that it is that peace of code

function waitForStorybook(browser) {
  return browser.executeAsyncScript(function (callback) {
    if (document.readyState == 'complete') return callback();
    window.addEventListener('load', function () {
      callback();
    });
  });
}

It turns out load event does not gets fired most of times for some reason and readyState stays in interactive state. I confirmed it by changing code like this

function waitForStorybook(browser) {
  return browser.executeAsyncScript(function (callback) {
    if (document.readyState == 'complete') return callback('readyState complete');

    window.addEventListener('load', function () {
      callback('load event');
    });

    setTimeout(function() {
    	callback('timeout. readyState: ' + document.readyState + ', loadEventEnd: ' + performance.timing.loadEventEnd)
    }, 50000)
  }).then(callbackReason => {
  	console.log(callbackReason) // Logs `timeout. readyState: interactive, loadEventEnd: 0`
  });
}

The issue happens only on headless Firefox on Linux. I tested the following machines

• Selenium Standalone on DigitalOcean droplet with Ubuntu 18
• Selenium Standalone on DigitalOcean managed Kubernetes with Debian 10
• Selenium Grid (Zalenium) on DigitalOcean managed Kubernetes with Debian 10
• Selenium Standalone on AWS managed Kubernetes with Amazon Linux 2

and reproduced bug on all of them.

I could not reproduced this bug with headless Chrome running on the same machines and Safari on other machine. Also sometimes load event does fire and everything works fine but it’s quite rare.

Steps to reproduce

1. Create machine with Ubuntu on any cloud service (I have tested on DigitalOcean)
2. Run official firefox selenium-standalone docker image: docker run -it -p 4444:4444 --shm-size=2g selenium/standalone-firefox:3
3. Add browser configuration to creevey config file

browsers: {
  firefox: {
    gridUrl: `http://${vmIP}:4444/wd/hub`,
    browserName: 'firefox',
    'moz:firefoxOptions': {
      log: { level: 'trace' },
      args: ['--headless'],
    },
  },
},

1. Run creevey npx creevey --ui
2. Wait for 1 minute
3. Result: selenium returns timeout error (with status code 500 for some reason, but response body clearly states that it’s a timeout error)

What to do

I’m not sure but it seems like we can use DOMContentLoaded event (and document.readyState === "interactive") instead of load event because we do not need subresources to be loaded (there is no fonts or images expect for favicon). It should also make creevey start a bit faster

[Perni1984]

@wKich adding a way to define a custom waitForStorybook function in the config or somewhere else would be great and would resolve some flaky tests / font loading issues I have with webkit (will post Issue + PR later).

[Perni1984]

@wKich well yes and no. FYI your font loading algorithm does unfortunately not work in webkit/safari due to an open bug there:

• https://bugs.webkit.org/show_bug.cgi?id=174030
• https://bugs.webkit.org/show_bug.cgi?id=217047
• https://github.com/web-platform-tests/wpt/pull/10025

At least I am experience flaky behaviour with font loading in webkit/safari 14.0, where the screenshots are taken too early (e.g. the webfont has not yet been loaded/applied).