Feature request: UI testing environment

Working on Wagtail’s admin UI, I find it unnecessarily hard to make sure that I take all of the variations in UI components into account. It’s a problem with all projects that have fairly big UIs, but is compounded in Wagtail by the fact that so much of the UI is dependent on what content has been created in the CMS, and on how the site has been set up. It also doesn’t help that Wagtail’s default dev environment is first meant to be a good demo of how Wagtail would be used on a small-scale website, rather than a dev playground (https://github.com/wagtail/bakerydemo/issues/175).

Having a more comprehensive UI “test” environment would help in a number of scenarios:

• Reviewing the existing UI, for example as part of a UX review, accessibility testing, compliance, or just when testing the impact of a specific change.
• Changing existing UI components, and reviewing changes. It’s really hard to know what impact a UI change might have, if you have to know how to use the CMS for bits of its UI to appear. This is particularly problematic when multiple people work on the same feature, and they all have to do the same error-prone setup on their local copies of Wagtail for them to test their changes meaningfully
• Setting up automated UI testing in continuous integration – accessibility testing (https://github.com/wagtail/wagtail/issues/4871#issuecomment-486648541), visual regression testing. These are impossible to set up if we don’t have a standardised test environment to build the test suites for.

Related: #3804, https://github.com/wagtail/bakerydemo/issues/175, https://github.com/wagtail/wagtail/issues/4871#issuecomment-486648541, https://github.com/wagtail/rfcs/pull/37

---

This was previously discussed at a Wagtail team meeting, and the conclusion there was that it should be done as a separate environment / project, rather than extending bakerydemo. It’s not entirely clear how this project could then be used in the Wagtail CI builds. Thoughts welcome!

[thibaudcolas]

@marteki interested in this

[jonnyscholes]

I’ve put up a proof of concept wagtail environment thats designed to be used by Wagtail developers and our CI. Yet to see how Travis goes with it, but you can run docker inside Travis so hopefully should be smooth sailing. Not completely happy with how we run the tests, but It Works™️.

Also verrrrry roughly mocked up using Pa11y’s actions along side one of your scenarios and that works really well! Loads more to do but it’s a start.

Be interested to get your thoughts on this @thibaudcolas given right now it would probably be mostly you and I using it, if we think it’s on the right track I’ll look into proposing how we could integrate it with our build pipeline (and start writing some tests for my JS refactoring 🎉).