"JavaScript heap out of memory" error occurs when build with lot of sub packages

Type of Issue

[x] Bug Report
[ ] Feature Request

Description

“JavaScript heap out of memory” error occurs when building with 15+ subpackages.

Here is console output and error reports

https://gist.github.com/johnny-mh/897f377424013e7f4b7048971c121b96

This project build successfully with angular7 versions

// angular 7 (works fine)
$ node_modules/.bin/ng-packagr --version
ng-packagr:            4.4.0
@angular/compiler:     7.0.1
rollup:                0.66.6
tsickle:               0.33.1
typescript:            3.1.3

How To Reproduce

Build library with many sub packages.

Expected Behaviour

Expect the build to work even with very many secondary-entrypoints

Version Information

$ node_modules/.bin/ng-packagr --version
ng-packagr:            5.3.0
@angular/compiler:     8.0.2
rollup:                1.16.2
tsickle:               0.35.0
typescript:            3.4.5

package.json

{
  "name": "@commerce-ui/ng-modules",
  "version": "0.0.0-PLACEHOLDER",
  "peerDependencies": {
    "@angular/common": "^8.0.1",
    "@angular/core": "^8.0.1",
    "@commerce-ui/util": "^2.4.0",
    "lodash": "^4.17.11"
  },
  "dependencies": {
    "@egjs/agent": "^2.1.5",
    "@egjs/flicking": "^2.5.1",
    "billboard.js": "^1.9.2",
    "@commerce-ui/legacy-lib": "0.0.6"
  }
}

[mistic100]

@johnny-mh please keep the issue open. I really think increasing the available memory is not good solution and ng-packagr should improve it’s memory usage.

[peterreisz]

I’ve investigated a little bit this issue and it seems the oldPrograms cache causing the memory issue.

With cache (original code): memory increase by about 300MiB / entry point and goes up to 6GiB at the end.

real    2m39,621s
user    3m47,822s
sys     0m5,627s

When I clear the oldProgram cache after the entry point is built: memory usage goes up to 1.5GiB at the end.

real    3m11,655s
user    5m54,193s
sys     0m4,500s

When I clear all the caches after the entry point is built: max memory usage is 1.2GiB and decrease to 300MiB at the end

real    3m39,834s
user    6m38,052s
sys     0m5,612s

Still working on the issue, but if you are willing to trade time for less memory usage then patch the entry-point.transform.js in your node_modules and run the ng-packagr directly with node --expose-gc node_modules/.bin/ng-packagr.

diff --git a/node_modules/ng-packagr/lib/ng-package/entry-point/entry-point.transform.js b/node_modules/ng-packagr/lib/ng-package/entry-point/entry-point.transform.js
index 4381e88..dc06044 100644
--- a/node_modules/ng-packagr/lib/ng-package/entry-point/entry-point.transform.js
+++ b/node_modules/ng-packagr/lib/ng-package/entry-point/entry-point.transform.js
@@ -15,6 +15,7 @@ const select_1 = require("../../graph/select");
 const transform_1 = require("../../graph/transform");
 const log = require("../../utils/log");
 const nodes_1 = require("../nodes");
+const ts = require("typescript");
 /**
  * A re-write of the `transformSources()` script that transforms an entry point from sources to distributable format.
  *
@@ -58,5 +59,15 @@ compileTs,
 writeBundles, writePackage, transform_1.transformFromPromise((graph) => __awaiter(void 0, void 0, void 0, function* () {
     const entryPoint = graph.find(nodes_1.byEntryPoint().and(select_1.isInProgress));
     entryPoint.state = node_1.STATE_DONE;
+    if (global.gc) {
+      entryPoint.cache.oldPrograms = undefined;
+      entryPoint.cache.sourcesFileCache.clear();
+      entryPoint.cache.analysesSourcesFileCache.clear();
+      entryPoint.cache.moduleResolutionCache = ts.createModuleResolutionCache(process.cwd(), s => s);
+      entryPoint.cache.rollupFESMCache = undefined;
+      entryPoint.cache.rollupUMDCache = undefined;
+      global.gc();
+      log.msg(`Clearing '${entryPoint.data.entryPoint.moduleId}' entrypoint cache (Heapsize: ${Math.round(process.memoryUsage().heapTotal / 1024 / 1024)} MiB)`);
+    }
 })));
 //# sourceMappingURL=entry-point.transform.js.map

You can use the patch-package to automatize the patching. It does not work without the manual gc call, the memory for oldPrograms cache will not be deallocated.