Inconsistent handling of script errors between the three engines

(This is with backstopjs@3.2.9)

I noticed an inconsistency with how the three engines handle script errors, with Puppeteer being the one more troublesome, imho.

I came up with this setup, which you can use to replicate the issue:

Three scenarios, one of which is designed to fail

"scenarios": [
  {
    "label": "Google",
    "url": "https://google.com"
  },
  {
    "label": "Google / Wait for wrong selector",
    "url": "https://google.com",
    "onReadyScript": "wait-for-wrong-selector.js"
  },
  {
    "label": "Google / Type",
    "url": "https://google.com",
    "onReadyScript": "type.js"
  }
]

For the two scenarios with the “onReady” script, there is a variation of the script for each of the engines

Casper

// casper/type.js
module.exports = casper =>
  casper.then(() => {
    casper.fillSelectors('form.tsf', {
      'input#lst-ib': 'foobarbar'
    });
  });
}

// casper/wait-for-wrong-selector.js
module.exports = casper =>
  casper.then(() => {
    casper.fillSelectors('form.tsf', {
      'input#lst-ib': 'foobarbar'
    });
  });

  casper.then(() => {
    casper.waitForSelector('.does-not-exist');
  });
}

Chromy

// chromy/type.js
module.exports = chromy => {
  chromy.type('input#lst-ib', 'foobarbar');
}

// chromy/wait-for-wrong-selector.js
module.exports = chromy => {
  chromy.type('input#lst-ib', 'foobarbar');
  chromy.wait('.does-not-exist');
}

Puppeteer

// puppet/type.js
module.exports = async puppet => {
  await puppet.type('input#lst-ib', 'foobarbar');
}

// puppet/wait-for-wrong-selector.js
module.exports = async puppet => {
  await puppet.type('input#lst-ib', 'foobarbar');
  await puppet.waitFor('.does-not-exist', { timeout: 1000 });
}

Now backstop reference and backstop test are run with all three engines, and these are the results that I’ve found

Casper

COMMAND | Executing core for `reference`
# ...
CasperJS:  CREATING NEW REFERENCE FILES
CasperJS:  Current location is https://google.com
CasperJS:  Capturing screenshots for desktop (1920x900)
CasperJS:  Ready event received.
CasperJS:  Current location is https://google.com
CasperJS:  Wait timeout of 5000ms expired, exiting.

Testing script failed with code: 1
# ...
      COMMAND | Command `reference` ended with an error after [9.965s]

With CasperJS the script error makes the entire command immediately fail and exit, meaning that BackstopJS can’t capture the remaining screenshots in the list.

Given that reference fails, there’s no point in running test given that it will fail as well and no report will be generated

Chromy

COMMAND | Executing core for `reference`
# ...
CREATING NEW REFERENCE FILES
9222 INFO >  BackstopTools have been installed.
CREATING NEW REFERENCE FILES
9223 Chrome v65 detected.
9223 INFO >  BackstopTools have been installed.
CREATING NEW REFERENCE FILES
9224 Chrome v65 detected.
9224 INFO >  BackstopTools have been installed.

# ...
      COMMAND | Command `reference` successfully executed in [10.396s]

Chromy marks the task as successful and doesn’t report any error whatsoever, although the screenshot for “Google / Wait for wrong selector” had not been generated

Running test gives the same output, but it’s marked as failed. This is report’s outcome

COMMAND | Executing core for `report`
      compare | Chromy error: WaitTimeoutError. See scenario – Google / Wait for wrong selector (desktop)
      compare | OK: Google backstop_default_Google_0_document_0_desktop.png
      compare | OK: Google / Type backstop_default_Google___Type_0_document_0_desktop.png
      # ...
      report | 2 Passed
      report | 1 Failed
      # ...
      COMMAND | Command `report` ended with an error after [0.177s]
      COMMAND | Command `test` ended with an error after [10.845s]

Now report reports on the WaitTimeoutError and it fails, showing in the html report the missing screenshot for “Google / Wait for wrong selector”

Puppeteer

COMMAND | Executing core for `reference`
# ...
CREATING NEW REFERENCE FILE
Browser Console Log 0: JSHandle:BackstopTools have been installed.
# ...
CREATING NEW REFERENCE FILE
Browser Console Log 0: JSHandle:BackstopTools have been installed.
######## Error running Puppeteer ######### Error: waiting failed: timeout 1000ms exceeded
# ...
CREATING NEW REFERENCE FILE
Browser Console Log 0: JSHandle:BackstopTools have been installed.

# ...

      COMMAND | Command `reference` successfully executed in [5.624s]

While it reports that something went wrong with a script (which is good for debugging), Puppeteer still allows for the entire test suite to run, and the report command is marked as successful

Running test gives the same output, and it’s marked as successful as well. This is report’s outcome

COMMAND | Executing core for `report`
      compare | OK: Google / Type backstop_default_Google___Type_0_document_0_desktop.png
      compare | OK: Google backstop_default_Google_0_document_0_desktop.png
       # ...
       report | 2 Passed
       report | 0 Failed
       # ...
      COMMAND | Command `report` successfully executed in [0.476s]
      COMMAND | Command `test` successfully executed in [6.424s]

There is no mentioning whatsoever of “Google / Wait for wrong selector” which at this point is simply ignored, report is marked as successful and the html report shows that everything is fine…but with only 2 screenshots instead of 3

---

While CasperJS is too eager to stop everything as soon as it finds an error, Puppeteer’s handling of script errors imho is worse as it simply marks the whole run as successful even if there are screenshots missing. If you have a big test suite (we have 120+ screenshots in our project), it’s very easy to trust the report and not realize that some screenshots might be missing.

I think the best behaviour should be a mix of Chromy + Puppeteer: show the script error in the output for debugging purposes, let the command run to completion and then mark test/report as failed and show the missing screenshot(s) in the report

[garris]

Yes. That is very specifically exactly what I was thinking.

[AkA84]

@garris no i’ve never approved any report in any of the tests i did as part of this issue, that’s why i’m reporting this as a bug. I think the behaviour should be the one displayed in your screenshot

and the one that Chromy has (with the due differences)

Running test gives the same output, but it’s marked as failed. […] Now report reports on the WaitTimeoutError and it fails, showing in the html report the missing screenshot for “Google / Wait for wrong selector”

otherwise imho this could be even worse than it was before: if before you knew the total number of scenarios you could quickly see that the number reported on was not right, now you can’t even do that ! You basically have to scroll down yourself and see if there are any errors reported (= you can’t trust the report)

And I’m not even considering the scenario where you are running backstop as part of a CI process where none of the two options are even possible 😕