StoryShots/Storyshots-Puppeteer is inconsistent and buggy with docker

Describe the bug We are trying to use Storyshots with puppeteer to take snapshots of our storybook and integrate it to out ci Jenkins pipeline, but are running into many problems when using it with docker, specifically these issues : Screenshots are inconsistent, primarily each time we run the tests with the exact same project and storybook hosted locally, we keep getting these errors related to mismatched resolution and diffs show all pages are slightly misaligned to the right.

We have 107 stories and somehow this is what happens when we run in docker

107 Fails + 107 Passes ?

To Reproduce My base chromium docker image, the rest is copying the project and running yarn test

FROM node:11.7.0-alpine
ENV CHROME_BIN="/usr/bin/chromium-browser"
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD="true"

RUN apk update && apk upgrade && \
    echo @edge http://nl.alpinelinux.org/alpine/edge/community >> /etc/apk/repositories && \
    echo @edge http://nl.alpinelinux.org/alpine/edge/main >> /etc/apk/repositories && \
    apk add --no-cache \
      chromium@edge=~73.0.3683.103 \
      nss@edge \
      freetype@edge \
      freetype-dev@edge \
      harfbuzz@edge \
      ttf-freefont@edge

RUN addgroup -S pptruser && adduser -S -g pptruser pptruser \
    && mkdir -p /home/pptruser/Downloads /app \
    && chown -R pptruser:pptruser /home/pptruser \
    && chown -R pptruser:pptruser /app

The docker image based on it.
Copies the storybook and runs the tests.

FROM node:11.7.0-puppeteer

WORKDIR /src
COPY --from= /src       .
COPY .                      storybook
RUN yarn install:ci:dev
CMD cd storybook && yarn test

imageSnapshots.js

import initStoryshots from '@storybook/addon-storyshots';
import { imageSnapshot } from '@storybook/addon-storyshots-puppeteer';

require('babel-plugin-require-context-hook/register')();

const waitBeforeSnapshotTime = 400;

// Hide all elements with classes containing these phrases
const ignoredElementsSelectors = ['MuiCircularProgress', 'MuiLinearProgress'];

const disableAnimationElements = ignoredElementsSelectors => {
  const animationSelectors = ignoredElementsSelectors.map(animationClass => `[class*='${animationClass}']`);
  const animatedPageElements = document.querySelectorAll(animationSelectors);

  animatedPageElements.forEach(pageElement => {
    pageElement.style.visibility = 'hidden';
  });
};

const storybookUrl = 'http://app-storybook:7071/';

const storiesToTestRegex = '^((?!.*?(Sketch|IFrame)).)*$';

const beforeScreenshot = page =>
  new Promise(resolve =>
    page
      .evaluate(disableAnimationElements, ignoredElementsSelectors)
      .then(() => page.setViewport({ width: 1920, height: 1080 }))
      .then(() => setTimeout(resolve, waitBeforeSnapshotTime)),
  );

const customizePage = page => page.setViewport({ width: 1920, height: 1080 });


initStoryshots({
  framework: 'react',
  suite: 'Image Storyshots',
  storyKindRegex: storiesToTestRegex,
  test: imageSnapshot({
    storybookUrl,
    beforeScreenshot,
    customizePage,
    chromeExecutablePath: process.env.CHROME_BIN,
  }),
});

Expected behavior The screenshots should consistent with the local ones especially since I only install puppeteer with addon-storyshots-puppeteer so there is no version mismatch.

The documentation does not make it clear that there might be differences between os’s and that results might be inconsistent.

Currently, for me, it is broken and I am trying super hard to make it work and to understand most the process required for it to work.

Thank you for reading my issue, I’m really hoping for a response and hope this will be helpful for others. 😃

[B3Kay]

I also got this problem. It renders different snapshots between computers.

[petermikitsh]

It is possible to get consistent screenshots across environments. Here’s how:

I ran into this issue and followed the strategy outlined in storybook-addon-image-snapshots (https://github.com/spring-media/storybook-addon-image-snapshots#testing-with-a-local-running-storybook-server). Two general principles I picked up:

• Generate your baseline snapshots and test snapshots using the same docker image. I’ve had success with browserless/chrome. You can mount your built storybook like so using -v.
• Always create snapshots against a built storybook (not when running start-storybook).

docker run -d --rm \
  -p 9222:3000 \
  -e "CONNECTION_TIMEOUT=600000" \
  -v $STORYBOOK_DIR/dist:/opt/storybook-static \
  browserless/chrome

Then, setup your tests with a custom puppeteer browser. The puppeteer driver will use Chrome from the docker container (above) exposed on port 9222.

Test Setup: storyshots.test.js

import initStoryshots from '@storybook/addon-storyshots';
import { imageSnapshot } from '@storybook/addon-storyshots-puppeteer';
import puppeteer from 'puppeteer';

const HOST_IP = process.env.HOST_IP;
let browser;

const testFn = imageSnapshot({
  storybookUrl: 'file:///opt/storybook-static',
  getCustomBrowser: async () => {
    browser = await puppeteer.connect({
      browserURL: (() => {
        // For CI env's: If running in docker inside docker, you'll need the IP address of the host
        if (HOST_IP) {
          return `http://${HOST_IP}:9222`;
        }
        return 'http://localhost:9222';
      })(),
    });
    return browser;
  },
  beforeScreenshot: async (page, { context: { kind, story }, url }) => {
    await page.evaluateHandle('document.fonts.ready');
    await page.setViewport({ deviceScaleFactor: 2, width: 800, height: 600 });
  },
  getScreenshotOptions: ({ context, url }) => {
    return {
      encoding: 'base64',
      fullPage: false,
    };
  },
});

// Override 'afterAll' so jest doesn't hang
testFn.afterAll = () => {
  if (browser) {
    browser.close();
  }
};

initStoryshots({
  test: testFn,
});

You’ll probably want 2 separate scripts: one for updating snapshots, the other for testing.

snapshots.sh

DOCKER_ID=$(docker run -d --rm \
  -p 9222:3000 \
  -e "CONNECTION_TIMEOUT=600000" \
  -v $STORYBOOK_DIR/dist:/opt/storybook-static \
  browserless/chrome)

npx --no-install jest --useStderr --update-snapshot

docker kill $DOCKER_ID

test.sh

DOCKER_ID=$(docker run -d --rm \
  -p 9222:3000 \
  -e "CONNECTION_TIMEOUT=600000" \
  -v $STORYBOOK_DIR/dist:/opt/storybook-static \
  browserless/chrome)

if [[ "$CI" ]]; then
  DOCKER_HOST_IP=$(/sbin/ip route|awk '/default/ { print $3 }')
fi

HOST_IP=$DOCKER_HOST_IP npx --no-install jest --useStderr
docker kill $DOCKER_ID