Feature request: add API to customise front-end HTML rendering of rich text features

Issue Summary

New feature: add an additional “rich text features” API to give developers full control over how the front-end HTML is created, for custom and also built-in features.

At the moment, unless the feature is a link with a linktype or an embed (images and actual embeds), we can only define how it is stored as HTML in the DB. This means that the feature’s “storage format” must be the same as its “front-end format”. This is too limiting – for example, it should be possible to add a class attribute to the feature’s front-end output without having it stored in the DB. It should be possible to store a strikethrough format as <s> tags but render them on the front-end as a more semantic <del>, should it make more sense for a given site.

Here is what the current APIs allows you to do:

features.register_converter_rule(
        'contentstate', 'blockquote', {
            [...]
            'to_database_format': {
                'block_map': {'blockquote': {
                    'element': 'blockquote',
                    # Those presentational attributes will be stored. Bad!
                    'props': {
                        'class': 'richtext__blockquote',
                    }
                }}
            }
        })

Ideally, there should be a separate to_frontend_html (or equivalent) to control the HTML on the front-end only. Note: it might not make sense to put this in register_converter_rule, since these are specific to a given converter (editorhtml for Hallo, contentstate for Draftail), whereas the new feature wouldn’t be.

Use cases

Here are the use cases we had when building WagtailDraftail, beyond custom links and embeds:

DRAFT_EXPORTER_BLOCK_MAP = dict(BLOCK_MAP, **{
    'unordered-list-item': {
        'element': 'li',
        'wrapper': 'ul',
        # Adding a class to rich text elements to make styling easier.
        'wrapper_props': {'class': 'list-styled'},
    },
    'header-two': 'h3',
    'intro': {
        'element': 'p',
        # Having a rich text format that's based on the same tag as other formats, + some classes.
        'props': {'class': 'intro'},
    },
})

Context

This is the last missing piece to bring Wagtail’s Draftail integration to parity with WagtailDraftail (and make that plugin useless, yay!).

It would also:

• Support what is requested in #3257 (and thus #1167).
• Help with mitigating #1214.
• Potentially replace the undocumented register_rich_text_embed_handler / register_rich_text_link_handler (#1094)

---

For the implementation, I’m not sure whether the existing handlers for linktypes and embeds can be repurposed / extended, or construct_whitelister_element_rules, or whether we need something separate. draftjs_exporter’s API is meant to support this kind of scenario, so one way to do this would be to do the contentstate from_database_format conversion, and then use Draft.js exporter to create the HTML (like to_database_format), but this might be overkill.

[loicteixeira]

Sorry about the repeated reference notification 👆 😰

So loicteixeira/wagtail@428f1081b53af083dec0bd3d99700039a5ce69df implements option 6 and here is the result of a quick benchmark:

Page	Notes	Measure	Original Time in ms	Patched Time in ms	Diff in %
/about/	Original Bakerydemo page. 1 rich-text block (body) 1 rich-text field (footer)	MIN	47.9	47.12	- 1.6
"	"	MAX	142.59	175.66	+23.19
"	"	MEAN	66.9296	71.9928	+ 7.56
/blog/bread-circuses/	Original Bakerydemo page. 1 rich-text block (body) 1 rich-text field (footer)	MIN	84.94	86.97	+ 2.38
"	"	MAX	190.38	307.29	+61.40
"	"	MEAN	106.972	136.044	+27.17
/new-page/	Manually created Standard Page. 10 rich-text blocks (body) 1 rich-text field (footer)	MIN	61.67	67.13	+ 8.85
"	"	MAX	267.2	175.4	-34.36
"	"	MEAN	91.086	102.6508	+12.69

Methodology

With debug_toolbar and template_profiler_panel to the bakerydemo project, run the following script twice, once with Wagtail unchanged, and a second time with Wagtail patched.

import re
from collections import defaultdict
from statistics import mean
from time import sleep

import requests
from bs4 import BeautifulSoup

data = defaultdict(list)

for endpoint in ("/about/", "/blog/bread-circuses/", "/new-page/"):
    for _ in range(25):
        response = requests.get(f"http://127.0.0.1:8000{endpoint}")
        soup = BeautifulSoup(response.text, "html.parser")

        panel_link_node = soup.find("a", {"class": "TemplateProfilerPanel"})
        panel_text = panel_link_node.find("small").text
        template_time = re.match(r'.*?(\d+\.\d+) ms$', panel_text).group(1)

        data[endpoint].append(float(template_time))

        sleep(0.25)

for endpoint, datapoints in data.items():
    print(f"Endpoint: {endpoint}")
    print(f"  Min: {min(datapoints)}")
    print(f"  Max: {max(datapoints)}")
    print(f"  Mean: {mean(datapoints)}")

[kaedroho]

I think we should replace the HTML representation of rich text fields with a JSON based representation based on StreamField and the new ParagraphBlock type described in https://github.com/wagtail/rfcs/pull/46

This would mean rich text fields would need a process for rendering to HTML, like StreamField. I think that could simplify the implementation of an API like this.

I also think that using JSON over HTML would also simplify the implementation of rich-text diffing, commenting, importers, and headless sites.

• Each paragraph, heading image, embed and list would be assigned a unique block ID which would be useful for rich text diffing and commenting
• JSON would be secure by default. If you are writing an importer and forget to whitelist the HTML, you could be exposing your site to potential XSS attacks. It would not be possible to represent a <script> tag in the JSON format so you won’t need to consider this
• I think most headless sites would have an easier time working with a StreamField-like JSON document over having to reimplement our custom HTML transformations
• We might be able to use StreamField blocks in rich-text some day (or make StreamField feel more like a rich text editor?)