Intermittent Failures

Hey all, We’re hitting intermittent bumps in our chromy-based backstop suite.

On a semi-regular basis we get false failures of this: compare | Chromy error: Error. See scenario – xxx xxx xxx xxx

Accompanied by a report that looks like this:

We’ve played with asyncCapture/compare limit, played with delay, but no dice. Sometimes it can be resolved by pkill Chrome, but generally we’re struggling to find the special sauce to solve them for good.

First question - have you encountered these intermittent failures? Second question - Is there a chromy/backstop flag we can set to investigate the error or capture some more context around this Chromy error?

[joseph-stano]

Alright, good news guys. That seemed to be our issue. We added wait(<selector>) to all of our DOM interactions and we’ve run our suite 15+ times with "asyncCaptureLimit": 20. All clear!

Here are the guts of our solution if you’re interested.

We have a helper script in which we introduced two functions for click() and type():

let chromy
exports.init = (chromyReference) => {
    chromy = chromyReference
    return this
}
exports.click = (selector) => {
    chromy.wait(selector).click(selector)
    return this
}
exports.type = (selector, text) => {
    chromy.wait(selector).type(selector, text)
    return this
}

And then in our onReadyScript:

const interactions = require('../interactions.js')
module.exports = function (chromy, scenario, vp) {
	interactions
		.init(chromy, vp)
		.click('.some-selector')
		.type('.another-selector', 'text to type')
}

A few chromy/backstop tweaks helped us debug this work:

• backstop: index.js:65 - log uncaught errors (this is generally helpful to catch errors in onReadyScript too)
• chromy: document.js - in reject(new WaitTimeoutError('...')) - we added the selector on which we were waiting. In a few of our scenarios we discovered we had bogus selectors that were never working in the first place, so our test was timing out.

Hope this helps others with the same issue!

[garris]

@kiran-redhat – thank you for tracking this down! I completely agree with your idea. We should add a log.error at line 319 which also includes the scenario label+viewport. This would make debugging MUCH easier for engineers!