[Brain storming] New language dependency system

Motivation

Our current language dependency system has a few problems:

• It is incompatible with ESM.

  The current system relies on load order to manage both require dependencies and optional dependencies. Optional dependencies simply cannot be expressed using ESM’s static import syntax.

• It has to be enforced by the user.

  Nothing stops our users from loading languages in the wrong order. We regularly get issues where people have this problem.

• It does not support third-party language definitions.

  Right now, all dependencies have to be declared in components.json. This makes it very hard for authors of third-party languages to integrate their languages. They are essentially forced to write languages with zero dependencies.

The first problem is huge because ESM is a goal for v2.0.

Goals

Primary goals:

• ESM-compatible.

  We have to be able to either write or create ESM source files. Source files have to be able to import each other to fulfill their require dependencies. (E.g. the JS language has to be able to import C-like.)

• Optional dependencies.

  The dependency system has to support optional dependencies. They are the reason this issue exists and they are necessary for Prism.

Secondary goals:

• Third-party authors should be able to “hook” into our dependency system.
• Making the Node hack unnecessary.

Non-goals

• Backwards-compatibility.

• Modify dependencies.

  Modify dependencies are not strictly necessary. Right now, Prism has 13 modify dependencies. 12 of those can be easily converted into optional dependencies. Only 1 modify dependency will be difficult to convert into an optional dependency but I’m sure that we can find a solution for that.

---

@LeaVerou @Golmote @mAAdhaTTah

[joshgoebel]

We might even want to go ES2018, because that added Unicode property escapes (e.g. \p{L}) which is huge for us,

[In Highlight.js] We use some ES2018 ([... but not {...), always keeping an eye on which browsers support which features… if something has been released for quite a while I’m generally not afraid of using it - generally we support “recent versions of green-field browsers”. The problem with some of the new stuff though is it’s a hard syntax error if you run into an old browser - which can sometimes break someone’s ENTIRE JS stack (when using a bundler)…

We started using \p a while back (perhaps slightly prematurely) and broke some very old Extended Service releases of Firefox, but after shipping it I didn’t see the value in rewinding it based on how tiny the affected user pool was - and those releases were going EOL soon anyways.

With v12 we’re planning to go ESM only (for Node) but for the browser we’ll still ship boring old CJS/global in addition to pure ESM modules. We still always resolve smaller dependencies at build-time… ie, our JS and TS modules are stand-alone - despite that in the source TS largely depends on JS. This is also true for our main library which in our case might be ~ 15 separate tiny JS files, but it compiles down to a one file monolith. Forcing the browser to import 15 different tiny modules makes no sense to me vs it being a build time concern.

Just my 0.02.

[LeaVerou]

I would be fine with only supporting very modern browsers with v2. Remember, cutting edge releases now, are the majority release in a year. It costs much more to refactor, whereas wider browser support is only a matter of time (and v2 will take some time to finish anyway). So I think only supporting the last 1-2 versions of current browsers when we start work on v2 is fine. I wouldn’t even consider browsers that don’t support type=module or import(), those are ancient history.

My understanding was that people use bundles to reduce HTTP requests and DNS lookups, not primarily for browser support.

I’m not quite sure what you mean with that. Like npm run build, minification included?

I believe he means import Prism from "https://prismjs.com/src/prism-core.js" (or local files), without having to make any bundles. But yeah, I suppose there’s always minification.