Many dynamicImports with FewestModules causes the Linker to GC itself to death

So this is a perplexing one.

I have a fairly big application that uses dynamic imports, and with that I use FewestModules for the production builds. However I added a new module recently and the linker started dieing basically from GC pressure.

I looked in jvisualvm and in the memory sample the number one object was a RedBlackTree in terms of total size. Unfortunately the code is private but I think it has something to do maybe the internal-id generation? And without seeing the code, the only difference from when the linker didn’t die and when it did, is that I added 2 dynamicImports. I do have sourceMaps on as well. And of course, the GCC is disabled.

Upping memory from 3g -> 4g didn’t make a difference.

Here’s the list of modules the commit before it started bombing (I removed the sourceMaps from the listing):

internal-0.js
internal-10608.js
internal-10938.js
internal-10.js
internal-12238.js
internal-12378.js
internal-1251730.js
internal-12.js
internal-1330878.js
internal-1338.js
internal-1345814.js
internal-1352956.js
internal-1388.js
internal-147062.js
internal-14.js
internal-1575752.js
internal-16.js
internal-1821838.js
internal-18.js
internal-2006.js
internal-2054544.js
internal-2085364.js
internal-2094766.js
internal-2097148.js
internal-2097149.js
internal-20.js
internal-2106.js
internal-212310.js
internal-215270.js
internal-22.js
internal-2368.js
internal-24.js
internal-252.js
internal-2662.js
internal-2676.js
internal-2678.js
internal-26.js
internal-28.js
internal-29240.js
internal-29260.js
internal-298.js
internal-2.js
internal-30.js
internal-310.js
internal-32.js
internal-333470.js
internal-34650.js
internal-34.js
internal-36.js
internal-370.js
internal-372.js
internal-38.js
internal-390.js
internal-394.js
internal-396.js
internal-398.js
internal-40142.js
internal-416.js
internal-418.js
internal-43178.js
internal-474788.js
internal-4.js
internal-52496.js
internal-603194.js
internal-66.js
internal-6946.js
internal-6.js
internal-740.js
internal-74.js
internal-758.js
internal-8726.js
internal-8.js
internal-93428.js
internal-951372.js
internal-951442.js
internal-996600.js
main.js

[gzm0]

More details: The problem here is that the module ID generation is O(2^(n +m)) where n is the number of public modules and m is the number of dynamic modules.

Now for n this is OK, because likely:

• The number of public modules is low.
• The connection graph is likely dense, so we’ll end up with all the combinations anyways.

However, for dynamic modules, the story is very different: A normal (i.e. reasonable) dynamic loading graph will likely look more like a tree with maybe some additional edges in between. Hence, it is sparse, so the ID generation will totally over-generate.

As a workaround, try to switch to SmallestModules; that should not suffer from this problem.

As how to fix it: We’ll need to find a way to generate module IDs in a somewhat stable manner. Because they are numbers, we might be able to use the same approach, but without actually calculating them. (i.e. dynamic module number k sets bit k in the module ID).

[dispalt]

Yep smallest fixed the problem, I can’t believe I left out that detail.