Proposal: image "variants" for better cross-platform testing

I’m trying to set up reg-cli for Copay, an Ionic app we build for iOS, Android, and desktop (macOS, Windows, and Linux).

I’m having some trouble avoiding false positives from very slight rendering differences between Chrome screenshots taken on macOS and those taken on Linux.

Almost everything is the same, but some font and shadow antialiasing seems to be different, and it’s causing false positives. There are also some platform differences causing problems (scroll bars, file upload buttons).

Here’s the comparison report from our CI: https://1043-18141084-gh.circle-artifacts.com/0/home/circleci/copay/test/visual/report.html

There are a few real issues mixed in with the false positives here, but I wanted to open this issue since it may be possible to solve a lot of these with improvements to reg-cli.

1: Antialiasing issues

For this case:

I really don’t know the best way to handle this issue. We’re currently using the reg-cli settings: --enableAntialias --thresholdRate 0.005.

Any recommendations for improving this? I’m afraid if I raise the thresholdRate more, we’ll start missing real issues.

It would also be helpful to understand what the diff is showing. What is yellow vs. red in the diff?

2: Scrollbars

I’m also trying to figure out the best way to handle platform rendering differences like scrollbars:

Our options here: A. Avoid any cross platform differences – e.g. render our own custom scrollbar on both platforms - we would prefer to use the native scrollbar, so this is not really an option B. Hide conflicting elements for e2e tests – we could inject CSS like * { ::-webkit-scrollbar { display: none; } just to take screenshots

• this is also not ideal, since we’d like screenshots to match the real application as best as possible C. Give reg-cli both a macOS and Linux version, so the test passes if either matches.

What are your thoughts on option C? If reg-cli could accept one or more variants for each image, I think this might also offer a better solution for our Antialiasing issues above.

Proposal

For images with lots of antialiasing problems, we can provide a version for both macOS and Linux.

Maybe a new command line option like:

--variantRegex the regex used to identify variants for each image. If omitted, (.*)-variant-[-0-9].

With the default setting, if my previous image list looked like:

test-image.png
test-image-variant-1.png
another-test.png
another-test-variant-1.png
another-test-variant-2.png

test-image-variant-1.png would match the default variantRegex, with a capturing group value of test-image. And both another-test-variant-1.png and another-test-variant-2.png would both have a captured value of another-test.

When testing a new test-image.png, reg-cli would first test against the existing test-image.png.

If it passes we’re done. If it fails, test against each variant for test-image (removing the .png to match the capturing groups).

If any variant passes, pass the test. If all variants fail, we use the closest variant for the diff. This allows us to specifically “pass” certain cross-platform differences, without missing real, unintentional changes.

Any thoughts on this idea?

[bitjson]

Update on this: after a lot of messy work trying to make our e2e system work better cross platform, I ultimately ended up switching the project to running the tests in a more controllable Docker container. You can read more about our process here.

So now we’re building everything on Linux, which solves most of our issues above. I don’t know if I would still recommend building a varientRegex option, I think it may be best to recommend projects take their screenshots in Docker or something similar.

And if you’re interested, here’s an example build:

BitPay build#1161 – reg-cli report Copay build#1162 – reg-cli report

I’m going to close this issue for now, but feel free to reopen it if you still want to pursue the idea.

[bokuweb]

@bitjson I see. It looks good:) But If --variantRegex not set, I think it would be better to disable the feature of the variant.