Test entire ecosystem upon each PR

Clear and concise description of the problem

New Vite releases often break the ecosystem without us knowning it.

E.g. 2.7.0-beta.x.

Suggested solution

Test the whole ecosystem upon each PR.

A plan I can see to work out:

• We create a new directory ecosystem/ living at Vite’s repository root.
• This directory contains bunch of submodules. E.g. ecosystem/vite-plugin-ssr/ is a git clone of github.com:brillout/vite-plugin-ssr.
• It is the responsibility of the ecosystem tool authors to maintain their entry in ecosytem/. E.g. I will regularly make a PR to update ecosystem/vite-plugin-ssr to the latest vite-plugin-ssr commit. I will also maintain a ecosystem/vite-plugin-ssr.test.sh script that executes everything needed to execute the vite-plugin-ssr test suite.
• The submobules are only checked out in GitHub Actions. (To not bloat Vite contributing.)
• The tests are only run in GitHub Actions. I.e. when running pnpm run test locally then all the ecosystem tests are ignored. (To not bloat Vite contributing.)

The most important thing here: it doesn’t affect Vite contributors in any way.

I don’t have capacity right now (focusing on Telefunc & Vike release) but I may have time work on it after Vike’s MVP release (can’t promise though; I forsee events that may likely swamp my agenda).

Alternative

No response

Additional context

This will:

• Make Vite rock-solid. E.g. vite-plugin-ssr and Telefunc have a lot of integration tests (Prisma, Cloudflare Workers (both miniflare & wrangler), Vercel, GraphQL, Redux, Vuex, …) so this will effectively ensure Vite doesn’t break the ecosystem at large.
• Make contributing 10x more comfortable. This gives contributors the comfort zone of knowing Vite contributions won’t break anything.
• Make developing Vite much more efficient. E.g. with our Vite 2.7.0-beta.x releases we have the painful situation that we don’t know when breaking changes were introduced. Catching errors early is a drastic improvement.

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn’t already an issue that request the same feature to avoid creating a duplicate.

[dominikg]

There is a lot of push back for testing the whole ecosystem upon each PR because of flaky test

My pushback against running it automatically on each PR is that it would be wasting resources. These are not quick feedback unit tests from vite’s perspective. Spending that amount of github action executions isn’t justified, esp when you add time spent evaluating and extra communication with vite contributors who don’t understand what these are about.

So I think that overall 17 minutes is more than acceptable, especially considering the benefits of running tests early rather than later.

17min for svelte is on the high side already, i don’t expect other integrations to top that. This is due to the fact that we have 2 test-suites, one for vite-plugin-svelte and one for sveltekit, both are executed 2 times, once with regular vite dependency and once with local vite. If needed we could parallelize a bit and bring it down to 10min.

Let’s try scheduled MWF + (beta) releases and see if we get caught off guard too many times. Then we can increase the frequency to nightly or even each commit/push to main (all filtered by changes to the actual vite package)

In order to reduce test flakiness, I think we should always pin a specific commit.

Pinning to a specific ref on external repo can be done inside the integration workflow as it depends on the project setup. This is one of the main advantages with separate workflows. They can be tailored to those projects needs

[dominikg]

i second (third?) Ben’s sentiment regarding git submodules and placing the burden of updating the submodule pointers on the ecosystem maintainers is just asking for trouble.

I’ll try to sketch what i’ve discussed and played with earlier - i’ll be using svelte as an example here bc i know the repos, the concept should apply to every other ecosystem project with github ci too (ideally also pnpm but other pm’s can be worked with i suppose)

1. use a separate repo ‘vite-integration-tests’ to hold actions running the tests
2. create an action for each integration vite + vite-plugin-svelte, vite+sveltekit, vite+x
3. in each action

• check out all required repos with https://github.com/actions/checkout into subdirectories. ref to check out could be an input param for the action defaulting to main
• build vite and link it into the other subdirectories (how to link exactly depends on the repo, pnpm link or edit package.json or whatever else works)
• build the subdirectories (hopefully all have a build script in package.json)
• run tests in the subdirectories (again hopefully all have a test script)

These actions can be triggered on a cron or manually (maybe setup parameterized manual and have a scheduler action that triggers them regularly - or maybe even event based if you can listen for commit events in other repos).