Cache is not cleared when `transformer` changes

What happens and why it is wrong

Without clean: true option, the plugin is catching the configuration even if changes at rollup config level are done (changing the implementation of a transformer implementation for instance)

Until verbosity: 3 was not set, it was not discovered cache could be an issue

Questions

• Is there any historical reason to set clean: false as default? It sounds like the default configuration should be without “”“magic”“” on top right?
• What do you think about adding a warning when cache is used so users notice about it with default verbosity level?
• I did not check how cache is implemented but, is it an expected behaviour to not clean the cache when there are changes on the transformers passed to this plugin or is a bug?

Environment

macOS catalina v10.15.4 node 14.2.0

Versions

• typescript: 3.9.3
• rollup: 2.10.5
• rollup-plugin-typescript2: 0.27.1

rollup.config.js

Not relevant

tsconfig.json

Not relevant

package.json

Not relevant

plugin output with verbosity 3

Not relevant

[ezolenko]

@ezolenko, for reference, do you have a preferred method of contacting you?

@agilgur5 yeah, github is sending too many emails all of the same kind (especially lately 😄), so I often miss pings here. I’ll play around with notification settings to limit those to PRs and direct pings, but you can also send direct email to zolenkoe at gmail.

Getting back to the very opening of this issue, given our current knowledge and those comments, it might also be a good idea to output a warning when a “dynamic” option that can’t be wholly cached is used

I think warning is not strong enough – maybe we should disable cache completely in those cases and output a warning about that (unless user provided extraCacheKeys option, we then leave everything up to the user). Then default behavior for users who don’t read warnings would be slower, but more correct compilation.

[agilgur5]

I think being thankful for this project, your time, and your effort is the minimum we can do as users.

💯 💯 💯 ❤️

They had a very ~wrong~ strange way to build their project I guess 🤷🏽.

I wanna say that rollup.config.js didn’t always exist and the Rollup API was first (I’m not 100% sure about that since I started using it once it already hit 1.0.0), so that might be historical context there. Also Grunt/Gulp/other JS-based task runners used to be pretty popular, and so too were the JS APIs used with them, so could have been something like that. Legacy code 🤷

transparent design limitations

My desire was to arrive at a solution transparent to the rollup-plugin-typescript2 users

Your proposal implies being “conscious” of which parts can be hashed incorrectly. This can be unintuitive for users not aware of all this conversation and the implementation details of this plugin.

Absolutely agreed. I think we’d all definitely prefer that.

Unfortunately, I don’t think we have much of any options, so I thought rather than just declaring this as blocked and unfixable, we could implement some workaround for it. Of course here in rpt2, clean: true, is always possible to disable the cache, but as that can be a significant deoptimization for larger projects, I thought that another workaround that leaves caching intact such as the proposed additionalCacheDependencies option, might the best we can get.

Also, as that’ll be an option listed in the docs, it can also act as a more of a hint to the user that the cache is not infallible (which the clean and objectHashIgnoreUnknownHack options suggest already, as well as the Contributing docs), and, in this case, can not only have bugs in the implementation, but can be literally unable to properly cache some things, like the dynamic behavior of functions.

Related: I left another comment just now in the linked object-hash issue, and with some even deeper investigation, this is practically impossible to do transparently. The best we could get is running a Babel macro on the rollup.config.js itself or an ESLint rule, but those are not transparent either. Not even Babel itself does that. From that perspective, an option like additionalCacheDependencies (similar to React’s useCallback hook’s dependencies) is significantly simpler to use, given only those options to compare against.

warnings

• What do you think about adding a warning when cache is used so users notice about it with default verbosity level?

Getting back to the very opening of this issue, given our current knowledge and those comments, it might also be a good idea to output a warning when a “dynamic” option that can’t be wholly cached is used, such as transformers in this case, or any other function options (like sourceMapCallback). That warning can point to the docs for additionalCacheDependencies which would reference this issue. That warning might dovetail well with the options schema checking feature (#312) I’d like to add as well.

We could add an info when using the cache in general, though that’s not the default verbosity level. At the default verbosity level, that might be output all the time for most users, so might be controversial (if you’ve ever tried to get all your logs to be silent in CI/CD, you know how annoying that can get; while setting verbosity: 0 can workaround that, that might not be ideal either).

• Is there any historical reason to set clean: false as default? It sounds like the default configuration should be without “”“magic”“” on top right?

Otherwise, anything else can be considered a cache invalidation bug, which, well, is known as one of the two hardest problems in Computer Science for a reason 😅

Caches are pretty ubiquitous in computing, e.g. DNS, CDNs, write buffers, L1/L2/L3 caches etc etc etc etc, and they default to “on” pretty often (and other libraries in the JS/TS ecosystem do as well), so I’m not sure this behavior is a departure from the norm in that sense, but I didn’t implement the defaults to know the original rationale. It would be a de-opt for larger projects by default too, as mentioned above.

The docs, contributing docs, issues, etc, do mention the cache, debug verbosity, etc, so I don’t think it should be altogether that surprising to users either. There aren’t altogether that many cache issues either, and I think in nearly all situations, users are aware they can use clean: true as a workaround too (but would prefer the cache be fixed of course).

But, of course, as with most things in software engineering, there’s trade-offs in either direction and I could honestly defend either as a default.

contributing

My interest in contributing is not as strong as before. I’m not in the company I hit this bug with, so I’m not actively working with rollup-plugin-typescript2 anymore.

Yup, totally expected that given the big gap of time, so no offense taken there. I could implement an additionalCacheDependencies option relatively quickly, but figured offering that up as a possible contribution would be better stewardship of the community, especially in this case where a community member had already expressed interest in the past 🙂

Still, I think this is a great opportunity to work and learn together with this draft PR.

Would you like me to draft something and keep discussing there?

That sounds like a plan to me! Though @ezolenko has final say, so don’t want to overstep there, but I imagine this feature would be welcomed. And it has a relatively small scope in terms of parts of the codebase it changes, which makes it more accessible and easier to approve for newer contributors. The most complicated part is probably getting the wording right for the docs and the naming of the option (in my opinion at least, obviously I know this codebase pretty well though, unlike most contributors) 😅

Separately, I’m also open to a PR (or PRs) with regard to additional warnings as well.