Issue: Very high memory usage on large projects

Hey there @sverweij 👋

This is mostly a notification of this issue I opened just now: https://github.com/webpack/enhanced-resolve/issues/229

Our usage of dependency-cruiser started running out of memory with the most recent versions.

I also used dependency cruiser on itself to do some memory profiling and it is somewhat representative of my use-case. According to chrome devtools, by the end of the run depcruise:report takes up about 270Mb (In my use-cases we’re up to Gbs 😅 )

I started to track down what was taking up so much memory and it turns out it was the cached file system from enhanced-resolver.

I tried removing the cache and just using their filesystem. The memory usage went down but the perf impact was big. In my codebase without the cache depcruiser takes x3 the amount of time to finish.

I dug deeper and realised that even though you pass a duration of 4000 to CachedInputFileSystem, the cache is never cleared. This is because even though it supports sync calls, the mechanism that checks whether the duration has been exceeded only run asynchronously. So on the case of depcruiser, given that it runs completely synchronously, the clearing mechanism is never called throughout the duration of the command.

I hacked up a version of enhanced-resolver that synchronously clears the cache every at the end of sync calls if the duration is exceeded and I don’t have memory issues anymore. The perf impact was relatively small, the command is only about 10s slower 🎉

So I opened this issue https://github.com/webpack/enhanced-resolve/issues/229 Hopefully, you’ll only need to update the version of this lib 🤞

Thanks again for your work on this 👏

Expected Behavior

Current Behavior

Possible Solution

https://github.com/webpack/enhanced-resolve/issues/229

(Ideally, you/we won’t have to rework everything to make async calls)

Steps to Reproduce (for bugs)

• Run depcruise:report with --inspect-brk
• Use Chrome devtools to get a heap snapshot before running and after its finished
• Most used up memory is coming from CachedInputFileSystem

Context

depcruiser runs out of memory on my CI

Your Environment

[sverweij]

@jomi-se there’s a dependency-cruiser@9.9.3-beta-3 out that uses ehr’s latest beta release, and which incorporates a way to tweak the cache duration; see the linked PR for details.

[sverweij]

hi @jomi-se

• First of webpack/enhanced-resolve#230 looks like great news. I’m for sure going to play with it tonight. First measurements indicate a 30% performance boost. 🎉
• For max memory usage I see a similar value as you do - just wonder why(1) it’s still that big (even with a timeout of 1000ms) and (2) why it stays allocated after the run (600Mb is better than 2.3Gb, but it’s still a lot …). I’ll put a filesystem.purge() after extraction is done - that should bring the memory allocated after the run back to normal proportions. When running dc against itself still ~50Mb remains instead of the 111Mb without the purge call.
• Based on the results you get with the timeout value I’ll re-assess the default with both smal (dc) and big-ish (react) projects. Based on what I’ve seen so far it’s likely to become smaller (~1s) rather than larger.
• I’ll definitely consider making the timeout configurable - either via the webpack config (by making the cached file system overridable) or in a separate option.

(and thanks for your elaborate research - it’s very helpful!)