[BUG] Chromium and webkit headless/head rendering differences

System:

• OS: Linux 5.4 Ubuntu 20.04.3 LTS (Focal Fossa)
• Memory: 1.12 GB / 15.45 GB

Binaries:

• Node: 14.18.0 - /usr/local/bin/node
• Yarn: 1.22.5 - /usr/bin/yarn
• npm: 8.5.1 - /usr/local/bin/npm

Languages:

• Bash: 5.0.17 - /bin/bash

Playwright env.:

• Playwright Version: 1.19.2
• Browser: All

The bug

Hi there! We are testing a kind of “analytics” web where some charts and diagrams are shown using amCharts library. The first approach is being made with snapshot testing.

The issue comes with head/headless option, as we develop with headless option set to false and the CI environment uses headless executions.

When the snapshots are made with headless option enabled and then we launch the tests in headful mode the tests in Chromium and Webkit are failing, the images are not equal. This doesn’t happen with Firefox, that makes identically component screenshots.

Evidences

We made a simple project that enters an amChart example and makes a snapshot of a pie chart component. The test consist on comparing a snapshot made with headless option set to true and then repeat the test in headful mode (and vice versa).

https://github.com/manuman94/playwright-head-vs-headless-rendering

Further, I tried to compare the images using pixelmatch library (which Playwright is using too I believe) and get the image difference. Here you can see the both images and the difference image: Headless:

HeadFul:

Difference:

Previous issues

There was an issue opened on Jan 4th 2021 (https://github.com/microsoft/playwright/issues/11177) regarding this issue, but as for today is closed and a pull request was created to solve the font family problems.

There is another closed issue (https://github.com/microsoft/playwright/issues/5371) but it seems that it was reporting a completely different problem.

Side notes

• Consequent screenshots in the same mode (headless or headful) are identical. This confirms that the amCharts library renders its components in the same way every time.
• The font family is not the only issue as the difference image shows that some pixels part of the chart are different too.
• Firefox does not have this difference.
• Is there any way of obtaining the result image using the matcher .toMatchSnapshot? The method signature tells that it returns a Buffer but I’m not able to obtain the difference image. The one I got is using pixelmatch function directly.

Thank you so much, greetings.

[aslushnikov]

Summary: browsers do render differently in headless and headed modes, so comparing headless screenshots with headful screenshots becomes complicated.

Rendering issues include:

• different font stacks for headless vs headful (attempts are made to fix in chrome)
• different antialiasing in headless vs headful (with the amchart example demonstrating the issue)
• scrollbars - hidden in headless and might be shown in headful.

Our options:

• can we align headless vs headful rendering across the board?
• shall we consider including headlessness to the upcoming screenshotsDir path?
  • if yes, then how?! do we do a resolver?
• shall we introduce an option to enable scrollbars in headless mode?

[gselsidi]

@aslushnikov

I’m noticing maxDiffPixels: 5, is not working… On ci there is a 1 pixel difference of the snapshot say 1x1 vs 1x2 and it fails. Tried 1000 pixels still fails

Expected an image 60px by 125px, received 60px by 124px…