`findFirstExistingPath` is extremely slow

We are currently evaluating using tsconfig-paths/register together with @babel/register for a pretty large, mostly js project (slowly migrating to ts)

I have found that both of these register hooks have an extreme overhead. For tsconfig-paths, most of that seems to come from a huge amount of filesystem accesses. Using it delays the startup of our app from ~5s to ~7s, and a profile shows quite clearly where the time is going:

I haven’t looked too much into what is exactly happening, but what I can say from the profile is that it apparently probes a ton of files before settling on which one to actually use. I will continue investigating and will update this issue with any findings.

[BJChen990]

Hi all, just run into the same problem that the findFirstExistingPath is extremely slow, but by looking at the frame graph, I think there’s also another problem that causes it extremely slow:

TL;DR

Using fs.statSync inside findFirstExistingPath when resolving path make it super slow for programs that uses require() inline. Suggest to fix with caching.

The problem

Using Typescript and postcards with webpack to build our own web application. Most of the part that includes require is super slow, like this part in html-webpack-plugin:

// setup hooks for webpack 4
    if (compiler.hooks) {
      compiler.hooks.compilation.tap('HtmlWebpackPluginHooks', compilation => {
--->    const SyncWaterfallHook = require('tapable').SyncWaterfallHook;
        const AsyncSeriesWaterfallHook = require('tapable').AsyncSeriesWaterfallHook;
        compilation.hooks.htmlWebpackPluginAlterChunks = new SyncWaterfallHook(['chunks', 'objectWithPluginRef']);
        compilation.hooks.htmlWebpackPluginBeforeHtmlGeneration = new AsyncSeriesWaterfallHook(['pluginArgs']);
        compilation.hooks.htmlWebpackPluginBeforeHtmlProcessing = new AsyncSeriesWaterfallHook(['pluginArgs']);
        compilation.hooks.htmlWebpackPluginAlterAssetTags = new AsyncSeriesWaterfallHook(['pluginArgs']);
        compilation.hooks.htmlWebpackPluginAfterHtmlProcessing = new AsyncSeriesWaterfallHook(['pluginArgs']);
        compilation.hooks.htmlWebpackPluginAfterEmit = new AsyncSeriesWaterfallHook(['pluginArgs']);
      });
    }

Reason

Every time the require is called, the Module._resolveFilename is called (which we replaced with our own version) and it uses an fs.statSync!!

For some Js project, there’re still quite a lot of places that put the require in the function body instead of extracting then out to the top of the script, which cause this problem. But still, it is not a good thing that the part called xxxSync is not cached as it would easily become slow.

Screenshots: The parts repeat and take quite a lot of times

It is stocked by the fs.statSync

Suggestion

I’m not familiar with the ts-config code base but I think we can introduce cache to the part https://github.com/dividab/tsconfig-paths/blob/master/src/register.ts#L82 maybe something like this:

// Some where as a file variable
const NOT_EXISTS = {};
const __matchPathCache = {};

// Inside register.js
Module._resolveFilename = function(request: string, _parent: any): string {
  const isCoreModule = coreModules.hasOwnProperty(request);
  if (!isCoreModule) {
+   if (request) {
+     found = __matchPathCache[request];
+   } else {
+     found = matchPath(request);
+     __matchPathCache = found || NOT_EXISTS;
+   }
+   if (found && found !== NOT_EXISTS) {
      const modifiedArguments = [found, ...[].slice.call(arguments, 1)]; 
      // Passes all arguments. Even those that is not specified above.
      // tslint:disable-next-line:no-invalid-this
      return originalResolveFilename.apply(this, modifiedArguments);
    }
  }
  // tslint:disable-next-line:no-invalid-this
  return originalResolveFilename.apply(this, arguments);
};

[jonaskello]

So monorepos is not typescript specific. It just means you have several packages in the same git repo. So you split your app in several npm packages with their own package.json even if you don’t intend to publish them because then you can import them using import * as Utils from "@myapp/utils". For this to work you need to create symlinks in the top-level node_modules that link to the code in each package. There are several tools that help with this, I would suggest reading up on yarn workspaces or if you are not using yarn then you can use lerna. Here is an exampe of using typescript with a monorepo (although in this example paths are still used but still it is a good example to start with). You can google “typescript monorepo” for more examples.

Once you have split your app into separate packages, you can make tsc build only the packages that have changed and thus get faster build times (this feature is known as “project references” or tsc --build). Here is one example repo for this approach. There is also this issue which may contain some useful links.

The basic idea is that all packages reference each other through regular node resolution, which is just look upward for node_modules/packagename. And since the symlinks are in place the packages will find each other without having to resort to relative file paths in imports.