UI architecture technical debt

Wagtail’s UI codebase is in bad shape. This isn’t really surprising – UI/CSS architecture in long-lived projects is a notoriously hard problem. It’s not hopeless either. But there is work to do 🙂

I’d like to use this as a tracking issue to discuss the problems this is creating, their causes, potential solutions, and hopefully gather consensus. In my opinion, a good outcome of those conversations would be to agree on a prioritised list of actions to be taken, that can then be worked on by different contributors in separate PRs. Sweeping rewrites are not on the agenda.

Edit: historically, the following content has been targeted towards CSS architecture in particular. But feel free to discuss any other UI maintainability issues (JS, HTML, fonts, SVG, etc) in this issue.

Problems

• Maintenance is hard and error-prone. UI regressions are frequent, and hard to correct.
• Trying to implement new UIs, there is a big bias towards creating new styles instead of repurposing existing patterns.
• Third-party plugins have no clear point of integration with Wagtail’s UI. Anecdotal examples.

Causes

This is a grab bag of code smells and high-level architectural issues.

• Main admin styles are kept in a few 100+ line files instead of being separated in components. (eg. _main-nav.scss, 479 lines)
• Too much nesting, generating overly-specific CSS selectors (eg. .nav-main .footer-submenu a).
• Styles rely on qualifying elements, generating overly-specific CSS selectors, making UI components harder to evolve to use different elements.
• No consistent naming convention. We are supposed to use BEM, but a lot of styles do not follow it, and it’s not enforced.
• Too many magic values. This is in particular problematic for spacing, where there is no consistency between different UI components.
• Too little architecture. There is a basic separation of components in different files, layouts with their page-specific styles, but it does not go far enough. (eg. for whatever reason all of the components that are specific to the page-editor layout have been defined inside of it.
• No clear boundary for plugin integrations.

Solutions

With #3773 as a foundation, I believe some of the above could easily be addressed by increasing what rules are enforced with tooling.

• Leverage more stylelint rules or plugins to enforce stricter guidelines
  • For qualifying elements, selector-no-qualifying-type.
  • For magic values, https://github.com/AndyOGo/stylelint-declaration-strict-value.
  • For naming,
    • https://github.com/kristerkari/stylelint-scss/blob/master/src/rules/at-function-pattern/README.md
    • https://github.com/kristerkari/stylelint-scss/blob/master/src/rules/at-mixin-pattern/README.md
    • https://github.com/kristerkari/stylelint-scss/blob/master/src/rules/dollar-variable-pattern/README.md
    • https://github.com/kristerkari/stylelint-scss/blob/master/src/rules/percent-placeholder-pattern/README.md
    • https://github.com/davidtheclark/stylelint-selector-bem-pattern
• Find and use an auto-formatting tool, like stylefmt or prettier.
• Find and use styles reporting tools to keep track of the number of colours, font styles, font sizes in use in the code.
• Use a stricter architecture than BEM, like ITCSS
• Make and mandate the use of a UI component library (#382)
• Expose Wagtail’s styles in the wagtail npm package so the variables and components can be reused in plugins.
• Define points of extensions for CSS with stable naming conventions.
• Develop and maintain a UI regression test suite to facilitate refactorings (https://github.com/thibaudcolas/wagtail-dev-tooling).
• Use pre-commit hooks to enforce linting when code is authored, rather than in CI.

Related issues, in no particular order

• z-index mess #1958
• Refactoring away from icon fonts #1224
• Streamfield UX issues have been left unaddressed in core because it is easier for people to override the styles than fix the problem directly in Wagtail #2325
  • https://github.com/heymonkeyriot/wagtailclearstream
  • https://gitlab.com/alexgleason/wagtailatomicadmin
  • https://github.com/springload/wagtailfacelift
• No documented way to override Wagtail’s CSS cleanly #2353
• ~Specificity hacks make it hard to customise Wagtail’s UI #2600~
• Theming capabilities would be cool #2943
• UI pattern library #3823
• Menu layout #4744

[thibaudcolas]

I’ve recently been considering which parts of this were still relevant, and whether to make a “2020” update to this all – the answer is that pretty much all of this still is relevant, and it’s felt quite heavily when doing major changes like those necessary for accessibility (#4199), or icons (#6107), which took many months’ more effort than if Wagtail’s front-end was better architected. We essentially waste a lot of time updating and testing patterns copy-pasted inconsistently across a big app, rather than being able to change components in isolation. I’ll focus the rest of the update on solutions rather than problems though.

Changes since the last update

• The SCSS refactoring (#6144) led by @nmorduch has been quite successful, moving more of the code to separate files. There’s more to do on that front: #5015.
• There was a similar JS refactoring piece started by @jonnyscholes in #5252, but that has stopped after a couple components.
• As part of the icons’ rework (#4821), @allcaps and @Scotchester took the time to refactor icons to a reusable component (via a template tag), which is essentially exactly what we would want to do for all UI components (or a similar approach with template partials includes, or Jinja2 macros, or React components)
• @lb- did a lot of good work with https://github.com/wagtail/eslint-config-wagtail and https://github.com/wagtail/stylelint-config-wagtail
• With #6105 (draft), we’re getting very close to removing Gulp and switching to Webpack only, which will eventually allow us to do more optimisations about what front-end code is loaded where.
• There’s now clearer plans to set up Prettier (#6059).

What to do next

The overall direction I would suggest to get out of this mess is to refactor the whole of Wagtail’s UI to be powered by a living pattern library, as suggested in https://github.com/wagtail/wagtail/issues/3804#issuecomment-422424260 and initially in #382.

Now working at Torchbox, I have a pretty clear sense of what this would look like: using https://github.com/torchbox/django-pattern-library, in conjunction with Storybook (https://github.com/torchbox/storybook-django), so we can have a living pattern library for both Django Templates, and React components. Both of those projects are relatively recent, but django-pattern-library has been in use for years internally at Torchbox, and Storybook itself has been around for years too.

The advantages of this approach are that:

• We’d be able to work on UI components in isolation from where they are used in Wagtail.
• We’d be able to test UI components in isolation – for accessibility issues and UI regressions in particular
• Since the components would be defined as isolated React components or template partials, they can be rendered outside of the Wagtail UI. This means we can test them without expensive browser automation to navigate around the Wagtail admin.
• Since each component would have “stories” to demonstrate its different variations, we wouldn’t have to set up expensive browser automation or hard-to-maintain fixtures in order to test specific parts of the UI.

With the Storybook version of this pattern library, we’d be able to leverage the Storybook ecosystem, which has well established tools for UI library smoke testing, visual regression testing, and accessibility checks. As well as integrations with design tools (Sketch, InVision, Zeplin). This could be a way for us to modernize Wagtail’s approach to design, and make it more open to third-party contributions (see https://github.com/wagtail/rfcs/blob/master/text/037-accessibility-support.md#3-make-universal-design-and-accessibility-part-of-wagtails-design-and-development-process).

⚓ Anchor link

What would this look like practically?

Components would be defined as folders, with all of the components’s code co-located:

.
├── accordion
│   ├── accordion.html
│   ├── accordion.py
│   ├── accordion.scss
│   ├── accordion.test.js
│   ├── accordion.ts
│   ├── accordion.yaml
│   ├── accordion_section.html
│   └── accordion_section.yaml
├── banner
│   ├── banner.py
│   ├── home_banner.html
│   ├── home_banner.yaml
│   ├── split_banner.html
│   ├── split_banner.scss
│   ├── split_banner.stories.tsx
│   └── split_banner.yaml
└── button
    ├── Button.stories.tsx
    ├── Button.tsx
    ├── button.html
    ├── button.py
    ├── button.scss
    └── button.yaml

The template partial, React component, stylesheet, test data, vanilla JS widget, per-component documentation, would all live in the same folder. For a Django templates integration, components that would make use of it could define their own template tag like our new {% icon %}.

Here is what component reuse would look like:

{# Django templates partials #}
{% include "ui/button/button.html" with label="Get started" variant="outline" color="primary" href="/" %}

{# Django templates tags #}
{% button label="Get started" variant="outline" color="primary" href="/" %}

{# React #}
<Button label="Get started" variant="outline" color="primary" href="/" />

{# Jinja macro #}
{{ button(
    label="Get started",
    variant="outline",
    color="primary",
    href="/",
) }}

{# Django templates partials #}
{% include "ui/banner/split_banner.html" with title="Stories from our Salesforce migration" description="Whisperly Ltd" image=image %}

{# Django templates tags #}
{% split_banner title="Stories from our Salesforce migration" description="Whisperly Ltd" image=image %}


{# React #}
<SplitBanner title="Stories from our Salesforce migration" image={image}>
    <p>Whisperly Ltd</p>
    <Button href="/">Get Started</Button>
</SplitBanner>

{# Jinja macro #}
{{ split_banner(
    title="Stories from our Salesforce migration",
    description="Whisperly Ltd",
    image=image,
) }}

{# Jinja macro with children #}
{% call split_banner(title='Hello World') %}
    <p>Whisperly Ltd</p>
    {{ button(label="Get started", href="/") }}
{% endcall %}

[patrick91]

Could be worth having a look at for the e2e tests http://cypress.io/ 😃