Performance Regression in 3.x

Current behavior:

Running 13 spec files in cypress@3.0.1 takes 152 seconds (including test compilation and execution)

Desired behavior:

Running 13 spec files in cypress@2.1 takes 79 seconds (including test compilation and execution)

Steps to reproduce:

My guess is that this regression is related to Cypress now running tests in isolation. There appears to be a lot of setup and teardown time between specs running. Is there anyway around this? Cypress taking twice as long to run the same test suite is surprising.

I’m not sure I can justify upgrading to 3.0 with this issue. It at least will make me think long and hard whether I can justify adding another test when it will be a significant impact to my CI run times.

UPDATE: My tests currently do an afterEach(() => { cy.screenshot() }) removing this cuts down execution time to 110 seconds.

Versions

cypress@3.0.1

[chrislopresto]

We’re seeing a similar performance regression from Cypress 2.1 to Cypress 3.1:

• 72 spec files
• Representative Cypress 2.1 run: 317 seconds
• Representative Cypress 3.1 run (no parallelization): 670 seconds

This would indicate a thoroughly unscientific, approximate ~5 second penalty per spec file. 🙂

With Cypress 3’s built-in parallelization (6 workers appears to be our sweet spot), we can improve the time of the overall test suite run to 96 seconds, so that’s fantastic. For better or worse that shifts our conversation to evaluating among a few options:

• Hold steady on Cypress 2
• Combine tests into fewer files. Presumably help unparallelized build time on free plan, hurt parallelized build time on paid plan.
• Upgrade to paid plan to use Cypress 3’s built-in parallelization
• Use our own parallelization scheme on Cypress 3 free plan

We’re happy to provide diagnostics and help dig in for troubleshooting as well. Cheers!

[jennifer-shehane]

@tomitrescak No there isn’t a flag. The browser relaunches in between each spec file so there is not any discrepancy between how runs run when parallelizaed (which would have each specfile in isolation completely) versus not.