Waiting for Storybook readiness does not work in headless Firefox
See original GitHub issueHello! I’m having difficulties running Creevey tests on Selenium with headless Firefox on Linux. Creevey is trying to start firefox and after a minite it fails with timeout
[CreeveyWorker]: Starting firefox:31362
[FAIL:31362] ScriptTimeoutError: Timed out after 60000 ms
at Object.throwDecodedError (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/error.js:517:15)
at parseHttpResponse (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/http.js:642:13)
at Executor.execute (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/http.js:568:28)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at mixin.execute (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/webdriver.js:731:17)
at runSequence (/Users/nitive/Work/s7_ds/node_modules/creevey/lib/server/utils.js:156:22)
at getBrowser (/Users/nitive/Work/s7_ds/node_modules/creevey/lib/server/selenium/browser.js:505:5)
at Object.worker (/Users/nitive/Work/s7_ds/node_modules/creevey/lib/server/worker/worker.js:196:19)
I started to debug what causes the timeout and found out that it is that peace of code
function waitForStorybook(browser) {
return browser.executeAsyncScript(function (callback) {
if (document.readyState == 'complete') return callback();
window.addEventListener('load', function () {
callback();
});
});
}
It turns out load
event does not gets fired most of times for some reason and readyState stays in interactive
state. I confirmed it by changing code like this
function waitForStorybook(browser) {
return browser.executeAsyncScript(function (callback) {
if (document.readyState == 'complete') return callback('readyState complete');
window.addEventListener('load', function () {
callback('load event');
});
setTimeout(function() {
callback('timeout. readyState: ' + document.readyState + ', loadEventEnd: ' + performance.timing.loadEventEnd)
}, 50000)
}).then(callbackReason => {
console.log(callbackReason) // Logs `timeout. readyState: interactive, loadEventEnd: 0`
});
}
The issue happens only on headless Firefox on Linux. I tested the following machines
- Selenium Standalone on DigitalOcean droplet with Ubuntu 18
- Selenium Standalone on DigitalOcean managed Kubernetes with Debian 10
- Selenium Grid (Zalenium) on DigitalOcean managed Kubernetes with Debian 10
- Selenium Standalone on AWS managed Kubernetes with Amazon Linux 2
and reproduced bug on all of them.
I could not reproduced this bug with headless Chrome running on the same machines and Safari on other machine. Also sometimes load
event does fire and everything works fine but it’s quite rare.
Steps to reproduce
- Create machine with Ubuntu on any cloud service (I have tested on DigitalOcean)
- Run official firefox selenium-standalone docker image:
docker run -it -p 4444:4444 --shm-size=2g selenium/standalone-firefox:3
- Add browser configuration to creevey config file
browsers: {
firefox: {
gridUrl: `http://${vmIP}:4444/wd/hub`,
browserName: 'firefox',
'moz:firefoxOptions': {
log: { level: 'trace' },
args: ['--headless'],
},
},
},
- Run creevey
npx creevey --ui
- Wait for 1 minute
- Result: selenium returns timeout error (with status code 500 for some reason, but response body clearly states that it’s a timeout error)
What to do
I’m not sure but it seems like we can use DOMContentLoaded
event (and document.readyState === "interactive"
) instead of load
event because we do not need subresources to be loaded (there is no fonts or images expect for favicon). It should also make creevey start a bit faster
Issue Analytics
- State:
- Created 2 years ago
- Comments:7
@wKich adding a way to define a custom
waitForStorybook
function in the config or somewhere else would be great and would resolve some flaky tests / font loading issues I have with webkit (will post Issue + PR later).@wKich well yes and no. FYI your font loading algorithm does unfortunately not work in webkit/safari due to an open bug there:
At least I am experience flaky behaviour with font loading in webkit/safari 14.0, where the screenshots are taken too early (e.g. the webfont has not yet been loaded/applied).