[Spike] Investigate increased memory usage with recent Firefox versions (98+)

Current behavior

When updating our internal CI image to run with Firefox 98, one of our existing builds started timing out/crashing due to a lack of resources. Example

I was able to trace this increase back to a commit and gather some metrics before/after it was merged:

Spec ran: runner/cypress/integration/reporter.errors.spec.js OS: macOS Monterey System: i9 / 16GB total RAM

Commit: 0bb655e379 |   |   | RAM (GB) |   |   |   – | – | – | – | – | – | –   | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Avg Firefox 99 | 2.55 | 2.53 | 2.59 | 2.68 | 2.43 | 2.556 Firefox 99 (numTestsKeptInMemory=0) | 1.39 | 1.18 | 0.953 | 1.32 | 1.29 | 1.2266 Chrome 101 | 1.55 | 1.69 | 1.74 | 1.59 | 1.6 | 1.634 Chrome 101 (numTestsKeptInMemory=0) | 1.13 | 1.12 | 1.14 | 1.15 | 1.12 | 1.132

Commit: 676fc97f46 |   |   | RAM (GB) |   |   |   |   – | – | – | – | – | – | – | –   | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Avg | % Increase Firefox 99 | 3.52 | 3.5 | 3.53 | 3.51 | 3.52 | 3.516 | 37.56 Firefox 99 (numTestsKeptInMemory=0) | 3.19 | 3.28 | 3.21 | 3.25 | 3.23 | 3.232 | 163 (❗) Chrome 101 | 1.63 | 1.58 | 1.66 | 1.59 | 1.59 | 1.61 | -1.5 Chrome 101 (numTestsKeptInMemory=0) | 1.27 | 1.28 | 1.26 | 1.27 | 1.26 | 1.268 | 12.01

To summarize, with 676fc97f46 (the origin feature branch merge) we start using a lot more memory with FF99. Chrome 101 stays relatively stable.

That spec file has 83 individual tests in it, and its snapshots can be quite large. So I also recorded runs with the numTestsKeptInMemory config value set to 0 to see what impact it had on memory usage. You can see that with 0bb655e379, setting that config property had a significant impact on FF99 and cut its usage in half; however, it had a minimal effect on 676fc97f46. It also had a decreased effect with Chrome 101 on 676fc97f46, with usage up 12% from the previous commit.

We should track down where/why this memory is being held.

Desired behavior

We don’t consume more memory than we have to when running with Firefox.

Test code to reproduce

1. Checkout cypress
2. Run yarn cypress:open
3. Run runner/cypress/integration/reporter.errors.spec.js in various browsers
4. Look at held memory after run completes

Cypress Version

9.6.0

Other

No response

[AtofStryker]

I was able to take a look into this. So it looks like the memory leak is coming from instances of Cypress’s primaryOriginCommunicator not being cleaned up between tests, but only in runIsolatedCypress tests. This is because the isolated tests recreate the iframe in which holds the cypress instance, but does not clean up the hanging references after the instance is no longer referenced. The primary domain communicator is only created once in regular cypress tests (such as the driver suite), which is why is looks like this leak is not present for normal use cases. I believe this issue is not impacting end users for this reason, only internal tests that leverage runIsolatedCypress in the 9.x develop branch.

reporter.errors.spec.js (~3,386 MB) with primaryOriginCommunicator enabled and all methods

reporter.errors.spec.js (~2,197 MB) with primaryOriginCommunicator commented out/disabled

reporter.errors.spec.js (~3,358 MB) with primaryOriginCommunicator enabled and one event present (leak detected)

I ran this against regular cypress tests in the driver as a control group to verify my hypothesis

driver check.spec.js (~196 MB) with primaryOriginCommunicator disabled

driver check.spec.js (~194 MB) with primaryOriginCommunicator enabled and all events

Since the 10.0-release merge is imminent into develop, and the runIsolatedCypress tests are no longer run or are removed, I decided to take applicable benchmarks for cy-in-cy tests, which only execute inside chrome currently. I ran the reporter.errors.spec.ts inside the app package, which is replacing the responsibilities of the runner package.

reporter.errors.cy.ts (~702 MB) with primaryOriginCommunicator enabled and all events

reporter.errors.cy.ts (~644 MB) with primaryOriginCommunicator disabled

Since this does not seem to be a user impacting users, and fixing it is likely going to not applicable in the 10.0 version of Cypress, my recommendation is that we likely do not need to fix this, but should likely revisit this issue once firefox is supported from within a cy-in-cy context.

[cypress-bot[bot]]

Released in 10.0.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to Cypress v10.0.0, please open a new issue.