Intermittent requests hanging, eventually crashing the tab/browser

Current behavior

This is an issue we’ve been seeing for some time intermittently in our CI pipeline, but we had a particularly clean / minimal example on Friday so this issue will specifically reference what we saw on that occasion.

We observed that one of our parallel runners had stopped emitting any output, suggesting that Cypress was stuck somewhere. Using VNC, we remoted into the display that Cypress uses to see what was going on, and what we saw was the browser spinning trying to load our tests. The browser was fully responsive, and refreshing the tab saw it getting stuck in the same place every time:

In the network tab, we could see that it was the XHR request to $BASE_URL/__cypress/tests?p=integration/ci/glean/tasks.spec.ts which was getting stuck. Chrome listed it as ‘Pending’ - I’ve attached some screenshots of what we could see in the network tab itself.

We were also able to reproduce in the Chrome console by firing off a manual fetch('$BASE_URL/__cypress/tests?p=integration/ci/glean/tasks.spec.ts') - the promise that was returned never completed. We did this about three times, before firing off one for a different spec file to see what would happen. Immediately as we did this, the browser completely crashed and we got a Javascript heap out of memory in the runner logs - see crashed output.txt.

We’ve been trying to narrow down this issue for some time now, and have held off raising a Cypress issue as we were concerned that it might be a regression in our own app. However, in this instance the problem was occurring before our site was even loaded, leading us to believe it’s a Cypress issue (specifically around the way that requests are proxied).

Desired behavior

No response

Test code to reproduce

Our cypress.json looks as follows, if it’s of any interest:

{
  "integrationFolder": "integration",
  "pluginsFile": "plugins/index.js",
  "screenshotsFolder": "screenshots",
  "videosFolder": "videos",
  "fixturesFolder": "fixtures",
  "supportFile": "support/index.ts",
  "chromeWebSecurity": false,
  "defaultCommandTimeout": 20000,
  "numTestsKeptInMemory": 0,
  "videoUploadOnPasses": false,
}

We are running Cypress in headed mode.

Cypress Version

9.5.1

Other

The browser in this case was Edge 100, but we’re seeing the same issues in Chrome as well.

[alyssa-glean]

Hey, just wanted to share an update on what we’ve been up to the last week or so.

I’ve just merged a PR that simplifies the test repo quite a bit more. There’s now just a single spec, which runs the same test in a loop 300 times. The test is a “failed login flow” - all it does is enter some credentials, hit ‘log in’ and expect an error to be shown.

One upside of this is that there’s no longer any need for any backend services, and so we’ve also added the local config file into source control (because there are no credentials needing to be wired anymore). This should mean that any collaborator can get up and running locally straight away 🎉

I’ve started experimenting with hosting our site in different ways to see if the problem remains or goes away. Currently our app is hosted in a k8s cluster behind traefik - I’m playing with a branch where we host it as a static S3 bucket instead on raw http. So far, I haven’t reproduced the hang on this branch (but would like to do more runs to be sure).

[alyssa-glean]

Just to give another update on this, we’re continuing to try and narrow down where the problem might be by gradually simplifying the test suite to see if the hang is still reproducible.

As part of this, we’ve made a public repo which runs the tests via github actions against a static branch that we’ll maintain for this purpose. It’s still reproducing the problem currently, see e.g. this build.

We’re happy to add folks from your side as collaborators if that sounds useful? Thinking it might be really helpful for you to have something to play with, and this way if you think of more diagnostics etc worth adding you could get the feedback more directly rather than having to go via us 😄