Compiler incorrectly caches module resolution if we use a custom ts.SourceFile & ts.Program cache

Bug Report / Question

At Google we use a “custom tsc”, which integrates with Bazel. It also implements something similar to --watch using persistent workers. The main difference is that the responsibility to watch for file changes lives in a separate process, which notifies a long running compiler process via IPC. The same compiler process is used to build many different projects. Projects may share dependencies and therefore read the same .d.ts files.

To make repeated builds fast, we use 2 caches:

• A ts.SourceFile cache lets us reuse ASTs across different projects
• A ts.Program cache lets use cache module resolution and type checking per project (by passing an oldProgram to ts.createProgram)

The TS compiler (used through the compiler API) sometimes incorrectly caches module resolution and reports “TS2307: Cannot find module”, even if the corresponding file exists.

🔎 Search Terms

TS2307, compiler API, cache, oldProgram

🕗 Version & Regression Information

• This is the behavior in every version I tried (4.0 - 4.7.4)

💻 Repro

We managed to create a minimal repro: https://github.com/frigus02/test-ts-oldprogram-moduleresolution

$ git clone https://github.com/frigus02/test-ts-oldprogram-moduleresolution.git
$ cd test-ts-oldprogram-moduleresolution
$ npm ci
$ node ./index.js

The repro has the 2 caches mentioned above and performs 3 builds, one after another, on the same project/list of files:

1. Successful build
2. Delete file /module/b.d.ts. Build fails with error “TS2307: Cannot find module ‘./module/b’”
3. Restore file /module/b.d.ts. Build still fails with the same error. But I would expect it to succeed again

🙁 Actual behavior

The 3rd build fails, seemingly because module resolution and/or diagnostics have been cached.

🙂 Expected behavior

The 3rd build succeeds.

🕵️ Investigation

I assume we’re violating an assumption in the compiler with the 2 different caches. In the repro, do you see any obvious misuses of the compiler API?

We found some solutions/workarounds. They either make the caches less effective or rely on @internal TypeScript APIs:

• Implement the @internal hasInvalidatedResolution() function in the CompilerHost.
• Only add dependency .d.ts files to the ts.SourceFile cache. Bazel should ensure that those are always without type errors.
• Remove one of the caches. It would have to be the ts.Program cache. The ts.SourceFile cache has a much bigger performance impact. We can’t afford to remove that.

From your perspective, is anyone of those the right thing to do?

[frigus02]

I had a look at the DocumentRegistry. If I understand correctly, it would give us 2 benefits:

• It caches source files by fileName & source file affecting compiler options. We could use getKeyForCompilationSettings to add this feature to our custom source file cache.
• It has a way to update a source file AST rather then re-creating from scratch after an edit.

In my test the document registry did not entirely fix the module resolution cache issue. That makes sense to me. It’s possible that builds use the same compiler options and only the available source files on disk change.

Maybe combining DocumentRegistry with the Watch Program API would work. I will need some more time to test that.

[sheetalkamat]

You can use DocumentRegistry for getting/creating source files since that handles when source files can be shared (depends on module resolution options as well as some other important options that if they differ between programs cannot share sourceFile)