[Feature]: Introduce a "lazy" haste-map implementation

🚀 Feature Proposal

This proposal is aimed at the react-native developer community. It introduces an experimental crawler to jest-haste-map, which should dramatically speed up metro start times and reduce resource consumption (CPU, memory, disk).

The new crawler is named lazy, and should reside next to the node and watchman crawlers. Unlike node and watchman, lazy doesn’t eagerly collect file metadata. Instead, it reads metadata on demand, as files are requested.

This breaks the design of Haste module loading, since there is no longer an in-memory listing of provided modules. In a react-native context, that’s ok, as almost no one there uses Haste.

The working theory is that the file graph generated on-demand by lazy (for bundling) will be much smaller than the entire monorepo file graph, and will therefore use fewer resources and take less time.

To validate this, we need a prototype implementation and a set of measurable experiments which look at CPU time, memory usage, disk reads/writes, time to create a bundle, and time to start a bundle server and deliver a bundle to a waiting app.

The metadata found by lazy should be cached, just as watchman and node data is cached.

lazy must support file-watching on files it has encountered, to ensure that metro (and other jest-haste-map clients) remain up to date.

Motivation

metro uses jest-haste-map to crawl the monorepo file-system. It gathers metadata on each file and watches for changes so it can keep the metadata up-to-date. The metadata includes things like a file hash and info about haste annotations within the file (e.g. @providesModule). The haste annotations allow a module to depend on another without using an explicit import or require statement.

metro crawls the file-system synchronously when it is launched, before it bundles or starts a bundle server. In large monorepos, crawling takes a long time and uses a lot of CPU, memory and disk resources. jest-haste-map uses caching to alleviate this. Even still, the cached data can take several gigabytes of memory. At this size, the haste map causes watchman to become unresponsive or crash at times.

Haste, as a module loading system, is not widely used by the react-native developer community. Yet, every developer pays the cost to run it on their machines and in CI loops.

metro recently landed a PR which lets developers replace the default haste map implementation with their own code. Using this, we can customize jest-haste-map to use the lazy crawler.

Example

No response

Pitch

It is a new crawler and therefore must be in jest-haste-map alongside existing crawlers.

[SimenB]

PR very much welcome adding an alternative crawler! 👍

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.