[Feature] HTML Reporter: UI Templating/Plugins

NB: 🗒️ https://github.com/microsoft/playwright/issues/11318 is related, but #11318 can be done today and is very specific/actionable, while this request is a bit longer term/extensible, but also decidedly more complicated and design-heavy.

Request

The HTML Reports are fantastic as is. However, now that we use them and receive them daily there are some customizations we’d love to make, but we don’t want to maintain an entire HTML reporter, so I’d like to propose a plugin/template system. With today’s HTML Report, there’s not really a place, except attachments to embed context/debug/trace information, and it’s always rendered as plain text.

We’d love to add our own company specific debug tools/links/etc. in a header block[^1] as well as define a custom rendering/visualization for some test attachments. Hopefully others would like this too 😃

Some additional examples:

1. Render clickable links back to the products/resources that the test created. When you’re debugging a an e2e smoketest that was done on a live environment, you can quickly jump to it from the report. This way, you don’t have to skim through a bunch of stdout/other attachments for an ID or URL to copy.
2. Render JSON attachments in a JSON Component Viewer. We attach big JSON blobs from our API responses, and looking at them in the report currently is less than ideal.

(Modulo the some different styling encapsulation) here’s wha the attachments section could look like with the user-authored plugins:

👋 I’m very interested in bringing this into reality in some form (see below for one quick prototype/idea[^3]), so please let me know your thoughts at both the idea and possible prioritization as it is a bigger lift and requires some design iterations on the public API. (I’m linking to code, but I’d like to focus on whether the idea/concept/constraints rather than an implementation/solution for now.) The prototype is not the implementation nor API I would necessarily propose, but it was fun to experiment and it would be interesting to iterate on an actual public API and implementation once that’s decided.

Thanks!

Prototype

I prototyped this a bit to see what it might feel like with some initial goals of:

1. Not exposing too much API surface area (i.e. not requiring the Playwright Team to maintain a React component library). The bigger the API surface, the more to document/test/troubleshoot/etc.
2. Let users write in React since many people working with Playwright, work with React, and React has lots of docs/support.
3. Continue producing a static, single-file final artifact bundle.
4. Bolted onto the current implementation 😃 so it can be experimental. Don’t require all the source artifacts used to produce basic report to be re-run through Webpacl. i.e. treat the user-land plugins as a 3rd-party UI lib from (the HTML Report view).
5. Auto-configured (i.e. don’t make users think about how to build/bundle React bits).

I arrived at an API that looks like the following as a prototype/conversation point[^2]:

⭐ See the following for an example of what the user would write in their project: https://github.com/rwoll/playwright/tree/feature/html-report-plugins/example-report-template

tl;dr: In their project folder, they write a index.(j|t)sx? whose default export satisfied an interface that has optional component definitions and an optional renderAttachment(a: Attachment) => JSX.Element function.

If they wanted to add a JSON component viewer for JSON attachments starting from that example, they could simply do something like:

$ npm i react-json-view

and then add the following to their renderAttachment function:

import ReactJson from 'react-json-view';
…
const renderAttachment: Components['renderAttachment'] = (attachment: TestAttachment) => {
  …
  if (attachment.contentType === "application/json" && attachment.body) {
    return <ReactJson src={JSON.parse(attachment.body)} />
  }
  return null;
};

export default renderAttachment;

See the PluginComponent for how the HTML Report renders the user-defined bundled components as an external/runtime lib/system. The user would know nothing about this. This component simply finds a potential plugin implementation and renders it while passing it some props, etc.

And see the following for how it’s all tied together to produce the final bundle: https://github.com/rwoll/playwright/blob/2a19ee183ecbda7bb513405c5fcb3d8b755c5c61/packages/playwright-test/src/reporters/html.ts#L383.

In reality, depending on what constraints we wanted to impose on the architecture, I’d implement this very differently. Right now, the implementation bundles and treats the user-land plugins as a “third-party JS UI” library, so the base HTML Report has been changed slightly so it can call into/render the “third-party” library that gets inlined as a distinct library. (Yes, that means React is in there twice 😛, but this was just for fun 😛.) However, if we didn’t pre-bundle the first report and bundled everything together at runtime, life is simpler, but that was less fun to try to get working 😃 Anyways, the implementation isn’t important atm 😄

[^1]: https://github.com/microsoft/playwright/issues/11318 would get us far for now though without all this complication! [^2]: Some parts of it I like, some parts are very meh. Plus, perhaps there are proper templating solutions out there for this kind of thing. This was just quick prototype to get some conversations/ideas going! [^3]: Warrants actual thinking and design hehe.

[dgozman]

Thank you for such a thorough write-up and prototyping. This is very exciting, but as you correctly say, not happening in the short term. Once we figure out the built-in features in the html report, we’ll have a good understanding of the extensibility perimeter.

[paulovpereira]

I also need a way to customize the HTML report for our company needs, but to be honest it would be already amazing if we could just create our own class that would extend PW HTML report class just like it’s explained here: https://playwright.dev/docs/test-reporters#custom-reporters

HTML reporter class needs to be exported for that to be possible.