End-to-end testing

Problem

Our current snapshot testing in packages/core only does so much. It does not:

• always make sure that the generated code is even valid JS
• that the generated components are valid in regard to the rules of their frameworks
• that the generated components have the features they are meant to have (interactivity, state management, styling, lifecycle hooks logic, etc.)

Another big issue is that we do not have a good idea of what features are supported by which generators at any given point. We currently have about 12 generators, and they don’t all support every single feature and edge case. It is unfeasible to manually gather that information and keep it up to date.

Ideal process

The ideal process I have in mind to accomplish this type of code coverage is to have an end-to-end testing pipeline like so:

• write a Mitosis project containing many components in packages/testing
• run it through the Mitosis CLI to generate a directory for each framework we support
• render every generated version of every framework on some server (N servers for N frameworks)
• For each framework, run the same integration test suite on each of its generated components in Storybook using a tool like Playwright https://storybook.js.org/addons/storybook-addon-playwright

To solve the issue of identifying feature support, we would then add some logic that performs all of these tests against every output, but allows for certain failures. We then need to generate a matrix of feature support (e.g. each row is a feature, each column is a framework) that gets printed in a markdown file in the repo (where this currently lives)

Example

So, if we have a clickable button, say packages/testing/src/button.lite.tsx.

1 - We would run mitosis build, and end up with:

• packages/testing/output/vue/button.vue
• packages/testing/output/svelte/button.svelte
• packages/testing/output/react/button.tsx
• packages/testing/output/solidjs/button.tsx
• etc.

2- We would load all of those using parallel servers (each in its own Github Action)

3- We would then have a test, say packages/testing/playwright/tests/button.ts. This test would execute on each github action against a different server.

4- We would finally compile all of the results and generate a matrix based on what passed and what failed.

Initial implementation

The quickest way to get an MVP of this idea is:

• Create a single, simple test (a div with text)
• Output it to just 1 framework (React to start would be good)
• import the generated components to a simple server that supports React
• run a playwright test that checks that the div has the expected text
• do all of the above in a Github Action task

Once that baseline is there, next steps are to:

• add more frameworks support: Vue & Svelte
• update testing to handle multiple framework tests in parallel

Once we have multiple frameworks, we can then start to generate a ton more tests, and increase framework support gradually!

[kylecordes]

Having used Storybook with various component frameworks, I’ve noticed that the config needed to deploy components in Storybook generally isn’t quite the same (certainly not bit-identical) to the config/code needed to use the components in the normal way, with no Storybook involved.

Therefore maybe think about a simpler variation?

Pile of mitosis components that exercise every feature of mitosis, and exercise any coding patterns that has revealed a bug in the past.

Compile those to all N frameworks.

Hand code the bare minimum for each of the frameworks to make it use those components - this should not be much.

Run the same Playwright test against the resulting N applications - I think this would cover all of the web targets. React Native and others may need separate support.

This would avoid having to step through the Storybook nuances for each target framework.

(Using any of the various build tools out there (Bazel, Nx, others) parallelization this should be relatively effort free.)

[samijaber]

Closed by https://github.com/BuilderIO/mitosis/pull/445 a while back! 🎉