Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[main branch] unexplained long hang/pauses in CI runners causing build timeouts

See original GitHub issue

We detected what appears to be a hang in the test suite when running .ci/scripts/test.sh -b "release" -t "apollo-server-express" "8" :

https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406

Relevant log entries:

<snip>

[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-926)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-927)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-928)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-929)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-930)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-931)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-932)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-933)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-934)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-935)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-936)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-937)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-938)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-939)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-940)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-941)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-942)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-943)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-944)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-945)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-946)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-947)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-948)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-949)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-950)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-951)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-952)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-953)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-954)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-955)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-956)[](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-nodejs%2Fapm-agent-nodejs-mbp/detail/main/95/pipeline/1406#step-4196-log-957)[2022-03-28T07:11:49.510Z] node_tests_1  | -- required packages ["graphql@^0.12.0","apollo-server-express@2.11.0"]
[2022-03-28T07:11:49.510Z] node_tests_1  | -- installing ["graphql@^0.12.0","apollo-server-express@2.11.0"]
[2022-03-28T07:11:59.500Z] node_tests_1  | -- running test "node test/instrumentation/modules/apollo-server-express.test.js" with apollo-server-express
[2022-03-28T07:12:01.408Z] node_tests_1  | -- running preinstall "npm uninstall express-graphql" for apollo-server-express
[2022-03-28T07:12:11.426Z] node_tests_1  | npm WARN nosup Unsupported engine for apollo-server-express@3.6.6: wanted: {"node":">=12.0"} (current: {"node":"8.17.0","npm":"6.13.4"})
[2022-03-28T07:12:11.427Z] node_tests_1  | npm WARN notsup Not compatible with your version of node/npm: apollo-server-express@3.6.6
[2022-03-28T07:12:11.427Z] node_tests_1  | npm WARN notsup Unsupported engine for graphql@15.8.0: wanted: {"node":">= 10.x"} (current: {"node":"8.17.0","npm":"6.13.4"})
[2022-03-28T07:12:11.427Z] node_tests_1  | npm WARN notsup Not compatible with your version of node/npm: graphql@15.8.0
[2022-03-28T07:12:11.427Z] node_tests_1  | npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.3.2 (node_modules/chokidar/node_modules/fsevents):
[2022-03-28T07:12:11.427Z] node_tests_1  | npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
[2022-03-28T07:12:11.427Z] node_tests_1  | 
[2022-03-28T07:12:11.427Z] node_tests_1  | -- required packages ["graphql@^0.12.0","apollo-server-express@2.10.1"]
[2022-03-28T07:12:11.427Z] node_tests_1  | -- installing ["graphql@^0.12.0","apollo-server-express@2.10.1"]
[2022-03-28T07:12:23.643Z] node_tests_1  | -- running test "node test/instrumentation/modules/apollo-server-express.test.js" with apollo-server-express
[2022-03-28T07:12:24.213Z] node_tests_1  | -- running preinstall "npm uninstall express-graphql" for apollo-server-express
[2022-03-28T07:12:34.199Z] node_tests_1  | npm WARN notsup Unsupported engine for apollo-server-express@3.6.6: wanted: {"node":">=12.0"} (current: {"node":"8.17.0","npm":"6.13.4"})
[2022-03-28T07:12:34.199Z] node_tests_1  | npm WARN notsup Not compatible with your version of node/npm: apollo-server-express@3.6.6
[2022-03-28T07:12:34.199Z] node_tests_1  | npm WARN notsup Unsupported engine for graphql@15.8.0: wanted: {"node":">= 10.x"} (current: {"node":"8.17.0","npm":"6.13.4"})
[2022-03-28T07:12:34.199Z] node_tests_1  | npm WARN notsup Not compatible with your version of node/npm: graphql@15.8.0
[2022-03-28T07:12:34.199Z] node_tests_1  | npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.3.2 (node_modules/chokidar/node_modules/fsevents):
[2022-03-28T07:12:34.199Z] node_tests_1  | npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
[2022-03-28T07:12:34.199Z] node_tests_1  | 
[2022-03-28T07:12:34.199Z] node_tests_1  | -- required packages ["graphql@^0.12.0","apollo-server-expres@2.9.16"]
[2022-03-28T07:12:34.199Z] node_tests_1  | -- installing ["graphql@^0.12.0","apollo-server-express@2.9.16"]
[2022-03-28T07:12:46.409Z] node_tests_1  | -- running test "node test/instrumentation/modules/apollo-server-express.test.js" with apollo-server-express
[2022-03-28T07:12:47.798Z] node_tests_1  | -- ok
[2022-03-28T08:48:52.336Z] Sending interrupt signal to process
[2022-03-28T08:48:58.013Z] Stopping docker_node_tests_1 ... 
[2022-03-28T08:48:58.014Z] Killing docker_node_tests_1  ... 
[2022-03-28T08:48:58.014Z] Killing docker_node_tests_1  ... done
[2022-03-28T08:48:58.014Z] Gracefully stopping... (press Ctrl+C again to force)
[2022-03-28T08:48:58.331Z] Sending interrupt signal to process
[2022-03-28T08:48:58.353Z] script returned exit code 2

Issue Analytics

State:
Created a year ago
Comments:13 (13 by maintainers)

Top GitHub Comments

1reaction

v1vcommented, Mar 30, 2022

Interesting, I’ve just enabled https://github.com/elastic/apm-agent-nodejs/pull/2629 to run a few times with a pre-docker upgrade image.

I’ll keep an eye for one day and then provide further details here before closing it.

0reactions

trentmcommented, Apr 13, 2022

THis is happening again: Screen Shot 2022-04-13 at 2 14 38 PM

For the latest one (build 114) it was the “Test / Node.js-14-async-hooks-false” build step that had an unexplained hang. From the log:

...
[2022-04-13T05:54:51.095Z] node_tests_1     | npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
[2022-04-13T05:54:51.095Z] node_tests_1     | npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated
[2022-04-13T05:54:51.095Z] node_tests_1     | npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
[2022-04-13T07:35:47.210Z] node_tests_1     | 
[2022-04-13T07:35:47.211Z] node_tests_1     | > dtrace-provider@0.8.8 install /app/node_modules/dtrace-provider
[2022-04-13T07:35:47.211Z] node_tests_1     | > node-gyp rebuild || node suppress-error.js
...

That delay was again (see earlier comment https://github.com/elastic/apm-agent-nodejs/issues/2624#issuecomment-1081258866) approximately 100 minutes:

> t2 = new Date('2022-04-13T07:35:47.210Z')
2022-04-13T07:35:47.210Z
> t1 = new Date('2022-04-13T05:54:51.095Z')
2022-04-13T05:54:51.095Z
> t2 - t1
6056115
> (t2 - t1) / 1000 / 60
100.93525

and in the same part of npm install. Could it be the binary build step for the ‘dtrace-provider’ npm package? That consistent 100 minutes suggests some hardcoded timeout.