Pre-bundled dependencies doesn't dedupe imports in external files

Describe the bug

When Vite pre-bundles a dependency, if the dependency’s entrypoint file imports an external file (e.g. Foo.svelte), imports in that external file aren’t deduped.

For example, given foo-library in node_modules with these files:

// shared.js
export const bar = {}

<!-- Component.vue -->
<script>
import { bar } from './shared' // pay attention
</script>

// index.js
export { bar } from './shared'
export { default as Component } from ',/Component.vue`

Vite will prebundle foo-library as .vite/foo-library.js:

// foo-library.js
export const bar = {}
export { default as Component } from '/@fs/node_modules/foo-library/Component.vue`

^ This is the problem, Component.vue will import bar from it’s relative './shared.js file, but .vite/foo-library.js has its own bar reference! This causes a lot of hard to catch bugs, especially for the Svelte ecosystem.

Reproduction

https://github.com/bluwy/vite-svelte-dedupe

(It’s a Svelte-specific repro but the general issue applies)

System Info

Output of npx envinfo --system --npmPackages vite,@vitejs/plugin-vue --binaries --browsers:

  System:
    OS: Linux 5.8 Ubuntu 20.10 (Groovy Gorilla)
    CPU: (4) x64 Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
    Memory: 4.49 GB / 11.59 GB
    Container: Yes
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 14.15.5 - ~/.nvm/versions/node/v14.15.5/bin/node
    Yarn: 1.22.5 - ~/.yarn/bin/yarn
    npm: 7.17.0 - ~/.nvm/versions/node/v14.15.5/bin/npm
    Watchman: 20210207.192227.0 - /usr/local/bin/watchman
  Browsers:
    Chromium: 91.0.4472.114
    Firefox: 89.0.1
  npmPackages:
    vite: ^2.3.8 => 2.3.8 

Used package manager: pnpm

---

Before submitting the issue, please make sure you do the following

• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn’t already an issue that reports the same bug to avoid creating a duplicate.
• Provide a description in this issue that describes the bug.
• Make sure this is a Vite issue and not a framework-specific issue. For example, if it’s a Vue SFC related bug, it should likely be reported to https://github.com/vuejs/vue-next instead.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.

[bluwy]

From some discussion lately in discord and https://github.com/sveltejs/vite-plugin-svelte/issues/125, it looks that auto excluding svelte (and probably marko, solid) libraries would be the way to go, since their compiled outputs are usually only finalised when evaluated in runtime, so they can’t really be pre-bundled ahead of time.

Issue

The issue now is that as @benmccann pointed out, CJS dependencies used by these excluded libraries (aka transitive CJS deps) wouldn’t work in Vite, since Vite doesn’t handle CJS in runtime, it expects everything to be in ESM already after prebundling, which is not the case for us. So to fix this, we have two ways:

1. Also optimize transitive CJS deps. This makes sure that the excluded library will use ESM code from its dependencies too, essentially eliminating any CJS code imported Vite dev server.

• Problem: This would be a breaking change for pnpm users unless they set shamefully-hoist=true or explicitly install the transitive CJS deps into the root project. This is because Vite’s optimizer would look for optimized libraries under /node_modules/ only, unless if it’s updated to look in nested node_modules and properly duped if needed when two excluded libraries use the same library of different major versions.

1. Handle CJS in vite dev server. So it loads the transitive CJS deps without issues.

• Problem: Could slow down dev server? (And probably more reasons I’ve not thought of)

Solution

An idea in mind is to work on no1, by altering the prebundling process to address the listed problem above. The altered flow should be like below:

1. For normal pure JS libraries, prebundle as usual.
2. For svelte, marko, solid (and even vue) libraries, vite will auto-exclude from optimization by default, but it’ll continue to scan for imports within the library (aka deep scan).
3. From the deep scan, for any imports found, vite will prebundle it as /node_modules/.vite/svelte-library__nested-cjs-dep.js. And make sure the library is resolved through /node_modules/svelte-library/node_modules/nested-cjs-dep, not /node_modules/nested-cjs-dep.
4. Import analysis should correct the imports in the excluded library to use the prebundled file (/node_modules/.vite/svelte-library__nested-cjs-dep.js).

In my mind, if I’m not missing out on anything, this would work nicely with X framework libraries OOTB.

There’s a small caveat however, that is these X framework libraries need to be in ESM since we exclude them from optimisation.

FAQ

Q: How do we differentiate pure JS libraries vs X framework libraries? A: The most robust heuristic is to check if that library has .svelte or .marko components, but that’s unneededly taxing on pure JS libraries. Otherwise a more lax heuristic is to check package.json for “svelte”, “marko”, “solid-js” dependencies (credit to @dominikg for the idea)

[benmccann]

There’s now an experimental option to prebundle .svelte files: https://github.com/sveltejs/vite-plugin-svelte/blob/main/docs/config.md#prebundlesveltelibraries