prerendering of 50k routes crashes on windows with ENAMETOOLONG

🐞 Bug report

What modules are related to this issue?

• aspnetcore-engine
• builders
• common
• express-engine
• hapi-engine

Is this a regression?

I think this was always broken.

Description

npm run prerender fails with spawn ENAMETOOLONG error when rendering 50k routes on windows. The root cause seems to be the OS limit on the command line length, which on windows is just 8000 bytes. I believe other operating systems are affected too but require more routes to break.

In my opinion it’s possible to fix this by adjusting _renderUniversal function in modules/builders/src/prerender/index.ts, which at the moment takes all routes assigned for a prerender process and passes them as command line parameters.

I have provided a sample fix below, but I wasn’t able to test it since I have never worked with bazel and couldn’t rebuild @nguniversal with this patch.

🔬 Minimal Reproduction

1. ng new hugesite
2. cd hugesite
3. ng add @nguniversal/express-engine
4. add a route { path: 'some-long-enough-route-name-to-make-the-length-realistic/:xyz', component: AppComponent } to src/app/app-routing.module.ts

import { NgModule } from '@angular/core';
import { RouterModule, Routes } from '@angular/router';
import { AppComponent } from './app.component';

const routes: Routes = [
  { path: 'some-long-enough-route-name-to-make-the-length-realistic/:xyz', component: AppComponent }
];

@NgModule({
  imports: [RouterModule.forRoot(routes, {
    initialNavigation: 'enabled'
})],
  exports: [RouterModule]
})
export class AppRoutingModule { }

1. update “prerender.options” section in angular.json with "numProcesses": 1, "routesFile": "routes.txt"

"prerender": {
  "builder": "@nguniversal/builders:prerender",
  "options": {
    "browserTarget": "huge-project:build:production",
    "serverTarget": "huge-project:server:production",
    "routes": [],
    "numProcesses": 1,
    "routesFile": "routes.txt"
  },
  "configurations": {
    "production": {}
  }
}

1. create mk-routes.js script to generate 50k random routes:

const fs = require('fs');

function genId() {
  const characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
  return [0, 1, 2, 3, 5, 6].map(() => characters.charAt(Math.floor(Math.random() * characters.length))).join('');
}

let routes = new Set();
while (routes.size < 50000) {
  routes.add('/some-long-enough-route-name-to-make-the-length-realistic/' + genId() + '\n');
}
fs.writeFileSync('routes.txt', Array.from(routes).join(''));`

1. run node mk-routes.js
2. run npm run prerender

🔥 Exception or Error

$ npm run prerender

> huge-project@0.0.0 prerender C:\Users\alex\Documents\huge-project
> ng run huge-project:prerender

√ Browser application bundle generation complete.
√ Copying assets complete.
√ Index html generation complete.

Initial Chunk Files               | Names         |      Size
main.60c1077a7d11e4aec8db.js      | main          | 212.36 kB
polyfills.94daefd414b8355106ab.js | polyfills     |  35.98 kB
runtime.7b63b9fd40098a2e8207.js   | runtime       |   1.45 kB
styles.09e2c710755c8867a460.css   | styles        |   0 bytes

                                  | Initial Total | 249.80 kB

Build at: 2021-04-24T09:27:44.892Z - Hash: 5a6f1bd8ac3117bbafa3 - Time: 10506ms
√ Server application bundle generation complete.

Initial Chunk Files | Names         |    Size
main.js             | main          | 3.16 MB

                    | Initial Total | 3.16 MB

Build at: 2021-04-24T09:27:45.746Z - Hash: 1ab7cc5ada205f8c62b4 - Time: 9657ms
× Prerendering routes to C:\Users\alex\Documents\huge-project\dist\huge-project\browser failed.
spawn ENAMETOOLONG
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! huge-project@0.0.0 prerender: `ng run huge-project:prerender`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the huge-project@0.0.0 prerender script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\alex\AppData\Roaming\npm-cache\_logs\2021-04-24T09_28_04_371Z-debug.log

🌍 Your Environment

$ npx ng version

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/


Angular CLI: 11.2.10
Node: 14.15.0
OS: win32 x64

Angular: 11.2.11
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, platform-server
... router
Ivy Workspace: Yes

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1102.10
@angular-devkit/build-angular   0.1102.10
@angular-devkit/core            11.2.10
@angular-devkit/schematics      11.2.10
@angular/cli                    11.2.10
@nguniversal/builders           11.2.1
@nguniversal/express-engine     11.2.1
@schematics/angular             11.2.10
@schematics/update              0.1102.10
rxjs                            6.6.7
typescript                      4.1.5

Suggested Fix

The fix idea is to measure the length of the command-line arguments and when they exceed 8000 bytes (or another system-dependent limit) to create another command. My example sollution depends on p-queue package to limit concurrency to numProcesses.

I’m not proficient with bazel, so wasn’t able to compile and test this code, so please consider it as pseudo code. I can work on refining the solution code since this problem is acute for my project, but I need some help and mentorship on how to compile and test patches for @nguniversal.

diff --git a/modules/builders/package.json b/modules/builders/package.json
index 57188620..8b2aaafe 100644
--- a/modules/builders/package.json
+++ b/modules/builders/package.json
@@ -35,6 +35,7 @@
     "guess-parser": "^0.4.12",
     "http-proxy-middleware": "^1.0.0",
     "ora": "^5.1.0",
+    "p-queue": "^7.1.0",
     "rxjs": "RXJS_VERSION",
     "tree-kill": "^1.2.2"
   }
diff --git a/modules/builders/src/prerender/index.ts b/modules/builders/src/prerender/index.ts
index 4cc19caa..c31400bb 100644
--- a/modules/builders/src/prerender/index.ts
+++ b/modules/builders/src/prerender/index.ts
@@ -13,6 +13,7 @@ import { augmentAppWithServiceWorker } from '@angular-devkit/build-angular/src/u
 import { normalize, resolve as resolvePath } from '@angular-devkit/core';
 import { NodeJsSyncHost } from '@angular-devkit/core/node';
 import { fork } from 'child_process';
+import PQueue from 'p-queue';
 import * as fs from 'fs';
 import * as ora from 'ora';
 import * as path from 'path';
@@ -118,41 +119,59 @@ async function _renderUniversal(
     }
 
     const spinner = ora(`Prerendering ${routes.length} route(s) to ${outputPath}...`).start();
-
     try {
       const workerFile = path.join(__dirname, 'render.js');
-      const childProcesses = shardArray(routes, numProcesses)
-        .map(routesShard =>
-          new Promise((resolve, reject) => {
-            fork(workerFile, [
-              indexHtml.replace('</html>', '\n</html>'),
-              indexFile,
-              serverBundlePath,
-              outputPath,
-              browserOptions.deployUrl || '',
-              normalizedStylesOptimization.inlineCritical ? 'true' : 'false' ,
-              normalizedStylesOptimization.minify ? 'true' : 'false' ,
-              ...routesShard,
-            ])
-              .on('message', data => {
-                if (data.success === false) {
-                  reject(new Error(`Unable to render ${data.outputIndexPath}.\nError: ${data.error}`));
-
-                  return;
-                }
-
-                if (data.logLevel) {
-                  spinner.stop();
-                  context.logger.log(data.logLevel, data.message);
-                  spinner.start();
-                }
-              })
-              .on('exit', resolve)
-              .on('error', reject);
+      const cmd = [
+        indexHtml.replace('</html>', '\n</html>'),
+        indexFile,
+        serverBundlePath,
+        outputPath,
+        browserOptions.deployUrl || '',
+        normalizedStylesOptimization.inlineCritical ? 'true' : 'false' ,
+        normalizedStylesOptimization.minify ? 'true' : 'false'
+      ];
+      const cmdSize = cmd.reduce((sum, c) => sum + c.length + 1, 0);
+      const renderCmd = (routesShard: string[]) => new Promise((resolve, reject) => {
+        fork(workerFile, [ ...cmd, ...routesShard ])
+          .on('message', data => {
+            if (data.success === false) {
+              reject(new Error(`Unable to render ${data.outputIndexPath}.\nError: ${data.error}`));
+              return;
+            }
+            if (data.logLevel) {
+              spinner.stop();
+              context.logger.log(data.logLevel, data.message);
+              spinner.start();
+            }
           })
-        );
-
-      await Promise.all(childProcesses);
+          .on('exit', resolve)
+          .on('error', reject);
+      });
+      const renderQueue = new PQueue({ concurrency: numProcesses });
+      const renderTasks = shardArray(routes, numProcesses)
+        .flatMap(routesShard => {
+          const batches = [];
+          let nextBatch = [];
+          let nextBatchChars = 0;
+          while (routesShard.length) {
+            const nextRoute = routesShard.shift();
+            while (cmdSize + nextBatchChars + routesShard[0].length >= 8000) {
+              if (nextBatch.length > 0) {
+                batches.push(nextBatch);
+                nextBatch = [];
+                nextBatchChars = 0;
+              } else {
+                throw Error('Based command line arguments are so large that do not allow even for a single route!');
+              }
+            }
+            nextBatch.push(nextRoute);
+            nextBatchChars += nextRoute.length + 1;
+          }
+          if (nextBatch.length) batches.push(nextBatch);
+          return batches;
+        })
+        .map(batch => renderQueue.add(() => renderCmd(batch)));
+      await Promise.all(renderTasks);
     } catch (error) {
       spinner.fail(`Prerendering routes to ${outputPath} failed.`);

[helix46]

Thanks @sierkov It links 3 google fonts and has a noscript. Size on disk is 1109 bytes I tested with these removed (533 bytes remaining) and prerender completed without error So to summarise, I get the ENAMETOOLONG error when:

• Win 10
• a single route
• index.html file size 1109 bytes

[angular-automatic-lock-bot[bot]]

This issue has been automatically locked due to inactivity. Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}