CT Performance Improvements: v10 slower than v9
See original GitHub issueEdit from @lmiller1990: this is getting worked on: https://github.com/cypress-io/cypress/pull/23104. Component Testing performance will be improved, initially released under an experimental flag to let use continue making incrementally performance updates. See latest comments for latest updates.
Edit: Try using the new experimentalSingleTabRunMode
flag in your component
property in cypress.config
. It is up to 40% faster for larger suites. Read more in the docs here.
Current behavior
After upgrading to the latest version, we have an issue with slowness when we run cypress run --component
. The tests themselves are not slow, but it seems there is some delay between specs runs as you can see in this video 👇 .
https://user-images.githubusercontent.com/6830426/174029644-aa31da35-ee35-4a04-8474-b0eab9a72869.mp4
We were not able to detect the exact issue, but we would appreciate any help 🙏
I was trying to run the tests with DEBUG
and this might be potentially the issue 👇
Desired behavior
The tests should run smoothly without any delay.
Test code to reproduce
We use cypress on public repo https://github.com/toptal/picasso.
We have cypress 10 on the current master.
yarn
yarn test:integration
for debugging it is faster to run
# these need to be run only once
yarn
yarn build:package
yarn test:setup cypress run --component
Cypress Version
10.1.0
Other
We use cypress with Happo for visual testing. The first idea was that it might be the problem since Happo didn’t adapt to v10 yet, but it is not the case, I have tried to remove it from the repo and it didn’t help.
Issue Analytics
- State:
- Created a year ago
- Reactions:10
- Comments:39 (23 by maintainers)
Top GitHub Comments
Summary
Ok, I looked into it. It turns out creating a new tab (and associated processes) does impact the performance. Here’s a branch with the 9.7.0 behavior, which is one tab -> run all specs in there, as opposed to the current implementation which is to create a new tab per spec.
There’s a build of this branch you can install for linux x64, get it here. There’s likely some bugs, but
npx cypress run --component
seems to work fine - if you are running linux, you could give it a try. I’ll get a more reliable build up for windows/mac soon.Note: these numbers are from 1 time runs - not the most scientific. To be more accurate, I’d run them all many times and take the average, but these are good enough to validate that opening a new tab is the primary reason we are slower.
Toy repo
For the run time: (+18% Cypress 9.7.0, +37% Cypress 10.3.0)
These numbers are a bit exaggerated, since I’m running 1 test per spec, which is not common. Normally, a single spec file will have 10s or 100s of tests. Either way, this experiment implies a ~300ms cost for creating a new tab, which is significant.
picasso repo https://github.com/toptal/picasso (run locally)
It’s 43% faster! I guess this mostly accounts for the 2x slowdown described.
Cypress internal component tests (local, no parallel, no video)
Run locally. I turned off video, which makes everything slower in a linear fashion, so it shouldn’t impact the relative differences. 30% is a solid improvement.
Discussion
I think it’s death by many a thousand paper cuts - just opening a new tab is not a single performance killer, but it looks like we do some cleanup between specs, and reset a bunch of things. I need to go deeper, but I suspect at least some of these things are absolutely necessary to guarantee spec isolation and determinism in E2E, which is why E2E has done the “new browser per spec” things since forever. I’ll look more into this, and if it’s necessary for component testing.
As shown on my toy repo with 100 of the space spec, as as predicted, the actual test execution time is identical, since your production code and specs hasn’t changed changed.
Next Steps
Thanks for the patience and assistance here everyone! I think we are on the right track. I’ll be focused on this until we have the performance issue solved.
There is definitely a slowdown here. Running your
picasso
tests, I see that:yarn test:setup cypress run --component
yarn test:setup cypress run-ct
It appears that with 10.x we do some cleanup work/reset some browser state between spec executions that is the bulk of the added time. This time increases linearly with the number of specs included with a given run. The debug log output from the 10.1.0 run is provided below.
Thanks again for the report. I’m going to direct this issue to the CT team to investigate mitigating factors.
10.1.0 debug output between spec run start and actually executing tests in the spec (click me)