[CI] occasional npm "Unexpected end of JSON input while parsing near ..." error

We have started seeing an issue in the master branch wherein npm seems to fail:

...
[2021-08-19T06:25:59.470Z] node_tests_1  | npm WARN deprecated b64@4.1.2: This module has moved and is now available at @hapi/b64. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
[2021-08-19T06:26:02.777Z] node_tests_1  | npm WARN deprecated vise@3.0.2: This module has moved and is now available at @hapi/vise. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
[2021-08-19T06:26:03.724Z] node_tests_1  | npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
[2021-08-19T06:26:04.297Z] node_tests_1  | npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
[2021-08-19T06:26:05.688Z] node_tests_1  | npm ERR! Unexpected end of JSON input while parsing near '...dependencies":{"sax":'
[2021-08-19T06:26:05.688Z] node_tests_1  | 
[2021-08-19T06:26:05.688Z] node_tests_1  | npm ERR! A complete log of this run can be found in:
[2021-08-19T06:26:05.688Z] node_tests_1  | npm ERR!     /tmp/.npm/_logs/2021-08-19T06_26_05_472Z-debug.log
[2021-08-19T06:26:06.260Z] docker_node_tests_1 exited with code 1
[2021-08-19T06:26:06.260Z] Aborting on container exit...
script returned exit code 1

[trentm]

The re-run jenkins builds above passed. In other words, this is an occasional error. I’m guessing it is a timing bug in npm. Perhaps in older versions of npm only. Perhaps exacerbated by our CI environment (perhaps because it is busy?).

I’m stopping active work on this issue for now. Let’s keep it around to use to track frequency of this error. If it maintains a high frequency, then we can dig into it more (considering the “solutions” mentioned above).

[trentm]

Note https://github.com/npm/cli/wiki/“cb()-never-called%3F-Exit-handler-never-called%3F-I’m-having-the-same-problem!” from the npm cli folks. I don’t have a lot of confidence that this will be solved completely in npm.

Looking at the ‘x’ for failed checks on https://github.com/elastic/apm-agent-nodejs/commits/master (not all of these are this error), indicates that while this is an occasional failure, it isn’t that occasional.

Moving to having a retry on the npm install step (perhaps with a npm cache clean --force before the retry) sounds reasonable. (Ideally this kind of change would also come with some metrics/stats on how often that retry is necessary, to know if/when we can ever remove having the retries. But we shouldn’t block on that.)