Cypress's Vite Dev Server exits when bundling cjs on the fly

Describe the bug

Disclaimer: @patak-dev told me to put this issue in here rather than in Cypress’s repo to start. I understand this issue is likely framework-specific. Happy to transfer it whenever necessary.

Vite x Cypress for Component Testing

👋🏻 Hello, I’m the author of @cypress/vite-dev-server, a package I wrote around March 2021 for use with the Cypress Component Test Runner. Until Vitest, using Cypress was practically the only way to test Vue/React components written and bundled with Vite.

The first work-around

In preparation for releasing Cypress 10, we’ve adopted Vite internally and have been heavy users of Vite over the last 8 months. Early on, we noticed that if Vite discovered a non-ESM module, it would reload the page and this would break Cypress when we used our Vite Dev Server plugin. It would fail to compile the cjs code and crash. We came up with a work-around:

We were able to ask users to use @antfu’s vite-plugin-optimize-persist and vite-plugin-package-config plugins to get a working dev server.

Using optimizeDeps kind of worked

Unfortunately, as we’re scaling up and integrating into other projects, we’ve found that those two plugins aren’t enough, and we’re seeing additional edge cases that are causing instability. We were treating one of the symptoms of Cy + Vite’s instability. The Cy dev server wasn’t crashing immediately, but there’s a deeper issue surrounding HMR and full page reloads. For example, in projects that use unocss, the way the styles are pushing onto the client causes another full page reload.

Reproduction

I’m currently blocked from merging a PR to add Cypress Component Testing for Vitest’s UI because of the flake. https://github.com/vitest-dev/vitest/pull/590

Cypress’s own monorepo has this issue and a 100% reproducible test case, but working in the Cypress monorepo can be a bit of a challenge. Here are some instructions…

1. Clone the main Cypress repo on the 10.0-release branch
2. yarn in the root of the repository
3. cd packages/app
4. rm -rf node_modules/.vite && yarn cypress:run:ct --browser chrome --headed --config video=false --spec **/SettingsContainer.cy.tsx

As a last resort…

One of the options we have is to switch to using vite build and statically serving the dist’d files. We don’t want to do this for a number of reasons: production parity, loss of HMR, and changes in compilation options (for example, Vitesse’s SSG requires <ClientOnly> on vite build, but not on vite dev. We want consistency between build environments).

Call for help

I’d really appreciate guidance and discussion around how to improve the stability of Cypress Component Testing with Vite.

Steps to reproduce

1. Checkout the PR on Vitest
2. pnpm install in the root of the project
3. cd packages/ui
4. rm -rf ./node_modules/.vite
5. pnpm ci

Afterwards, you can check the packages/ui/cypress/videos directory to see a video of the failed test.

When you re-run the test, it will pass (some of the time) because the packages have been prebundled. To reproduce, make sure you’re running rm -rf ./node_modules/.vite before running pnpm ci

Reproduction

I wasn’t able to minimize the reproduction further than the already broken PR in Vitest: https://github.com/vitest-dev/vitest/pull/590

System Info

System:
    OS: macOS 11.4
    CPU: (16) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 3.06 GB / 32.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.17.0 - ~/.nvm/versions/node/v14.17.0/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 6.14.13 - ~/.nvm/versions/node/v14.17.0/bin/npm
  Browsers:
    Chrome Canary: 99.0.4840.0
    Firefox: 95.0
    Safari: 14.1.1
  npmPackages:
    @vitejs/plugin-vue: ^2.0.1 => 2.0.1 
    @vitejs/plugin-vue-jsx: ^1.3.3 => 1.3.3

Used Package Manager

pnpm

Logs

😅 if there were instructions for how to turn --debug logs on from within the JavaScript API, that’d be great.

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn’t already an issue that reports the same bug to avoid creating a duplicate.
• Make sure this is a Vite issue and not a framework-specific issue. For example, if it’s a Vue SFC related bug, it should likely be reported to https://github.com/vuejs/vue-next instead.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[JessicaSachs]

TL;DR so far

👋🏻 It’s been a big struggle to get the environment up. That’s what I spent most of my day doing – working toward the goal of being able to prototype the suggestions that @patak-dev and I spoke about.

What I need is a test fixture or project/page that has many optimizeDeps cache misses and ends up responding with a lot of 408s. Ultimately, triggering this block of code:

https://github.com/vitejs/vite/blob/fb7ba5353bb35d41f2ec2c90b6b8e5fecd508d07/packages/vite/src/node/server/middlewares/transform.ts#L66-L94

Next steps

I will work on this tomorrow to try and simplify the reproduction further and move it out of Cypress-land. If someone has a simplified project I can use that triggers the failure state I’m talking about, that would be super helpful.

For a better demonstration of the symptoms we’re seeing, I’ve attached a video. It’s split into two parts. Essentially we’re getting a ton of 408’s, reloads, and marvelous failures.

The initial failure. This has like 4 reloads.

https://user-images.githubusercontent.com/2801156/151488655-5667e289-1afc-44bc-9780-6cf2796090de.mov

Success when reloading

https://user-images.githubusercontent.com/2801156/151488882-cdd38ec7-c8e3-4931-b4ba-3c444e0e2e1a.mov

This only really happens for the first build of the first page for a user’s particular machine when the project is large enough

So, basically, most CI runs.

Project Info More details about the project under here, including information for where to find the Vite configs and plugin code written.

The Vite config used for the page in the video is here in the Cypress App – the web frontend, but most of the config is within the frontend-shared package’s Vite config.

Package Arch

The Cypress + Vite Dev Server code lives here. The architecture is stupidly simple, we create a Vite dev server from the JavaScript API and serve each spec file as a “page” in Vite.

Everything functions fine outside of the full-page refresh behavior for the first spec.

Thanks for the help so far. I’m around to Discord for sync comms. My plan is to keep this issue updated at least daily because solving this is really critical to Cy and Cy’s upcoming release.

Cheers, Jess

[JessicaSachs]

Final update for now.

@patak-dev asked me to keep this issue open as a placeholder to discuss an architecture that does not have the below flaws and resolves all dependencies AND transient dependencies such that incremental reloads are not necessary.

Update on the previous approach:

1. We’ve found the fatal flaw in the above strategy which is that when resolving dependencies, shared transient ones may resolve to a different version. (This is what Evan told me in October, lol, I just didn’t fully understand how limited we were until now.)
2. If we were to push up a new version for a transitive dep, we would have multiple versions of a dependency running within the same page context.
3. Having multiple versions of certain packages running at the same time can cause very critical application errors (e.g. React can’t run multiple versions on one page).
4. Additionally, without doing a full-page reload, any side effects would not be cleaned up.

Therefor, unless we know all of the transient dependencies for all entries, we cannot safely prevent a full page reload.

Cypress’s decision is to use vite build + vite serve in CI. This will reliably solve the issue for fresh machines that have never run the user’s application before.

For local development, we will continue to use vite dev to benefit from all of the debug and development utilities that plugins like WindiCSS provide to users.

The surface area of risk for local development is restricted to users who have:

1. Never run their application before
2. Never run their tests before
3. Deleted their cacheDir

In all of these local-only edge cases where a user has a completely clean cache, a simple Cmd + R will fix this issue.

Timeline

1. After the Cypress v10 launch, we will improve the DX for users developing locally with cypress open.
2. Immediately (today), we will make @cypress/vite-dev-server utilize vite build + vite serve when running Cypress in run mode (AKA, CI). This will fix the majority of the instability for all versions of Cypress that can run Component Testing v7+.

Future

If the Vite team ever has a way to turn off automatic reloads, that would allow us to create a more consistent experience between dev and prod.