Massive decrease in performance after updating from `v9.0.3` to `v9.1.0` (test executions are very slow)

Prerequisites

• Checked that your issue hasn’t already been filed by cross-referencing issues with the faq label
• Checked next-gen ES issues and syntax problems by using the same environment and/or transpiler configuration without Mocha to ensure it isn’t just a feature that actually isn’t supported in the environment in question or a bug in your code.
• ‘Smoke tested’ the code to be tested by running it outside the real test suite to get a better sense of whether the problem is in the code under test, your usage of Mocha, or Mocha itself
• Ensured that there is no discrepancy between the locally and globally installed versions of Mocha. You can find them with: node node_modules/.bin/mocha --version(Local) and mocha --version(Global). We recommend that you not install Mocha globally.

Description

We execute Selenium tests with mocha. After updating from v9.0.3 to v9.1.0 yesterday, the test execution performance decreased massively:

• A test batch that prior to the update took ~9 minutes, after the update takes ~25 minutes
• A test batch that prior to the update took ~12 minutes, after the update takes ~28 minutes

Steps to Reproduce

Use mocha v9.1.0 -> performance is bad Downgrade mocha to v9.0.3 -> performance is normal again

Expected behavior: [What you expect to happen] Performance in v9.1.0 to equal the performance in v9.0.3.

Actual behavior: [What actually happens] Performance is decreased massively after updating.

Reproduces how often: [What percentage of the time does it reproduce?] 100% Confirmed through multiple executions on “fresh” Github Action machines. Confirmed the issue goes away after downgrading to v9.0.3 and executing the tests again.

Versions

• The output of mocha --version and node node_modules/.bin/mocha --version: v9.1.0 (after update) and v9.0.3 before
• The output of node --version: 14.17.5
• Your operating system
  • name and version: Ubuntu 20.04.2
  • architecture (32 or 64-bit): 64
• Your shell (e.g., bash, zsh, PowerShell, cmd): bash
• Your browser and version (if running browser tests): N/A
• Any third-party Mocha-related modules (and their versions): ts-mocha v8.0.0, mocha-junit-reporter: v2.0.0, mochawesome: 6.2.2
• Any code transpiler (e.g., TypeScript, CoffeeScript, Babel) being used (and its version): typescript v4.3.5.

Additional Information

This is how we execute the tests: executetestbatch_1.js

const Runner = require("mocha/lib/runner");
const { filterSuite } = require("./executetestbatchhelper");

const originalRun = Runner.prototype.run;

const batchID = 0;
const numberOfGroups = 3;

Runner.prototype.run = function (done) {
  // Filter tests to only include Tests in current batch
  this.suite.suites.forEach((suite) => filterSuite(suite, numberOfGroups, batchID)); //// <------ This just filters to a subset

  // Execute tests
  originalRun.call(this, done);
};

The command line is basically "test:e2e:pipeline:batch1": "npm run test:e2e:pipeline -- -- -- --require ./src/test/executetestbatch_1.js", <-- gets executed "test:e2e:pipeline": "cross-env CI=true npm run selenium -- -- --bail --reporter mochawesome --reporter-options reportDir=testreports,reportFilename=SeleniumTestReport", "selenium": "cross-env HOTRELOAD=no ts-mocha \"./src/**/*.roxselenium.ts\" --exit --recursive --timeout 120000 --retries 2"

I unfortunately don’t know where to start to further investigate what is causing this performance decrease. If anyone could give me any hint, I would appreciate it. Since we isolated that the issue really comes from the update (and is good again after downgrading), I guess either something in mocha changed or there is now an incompatibility (/performance decrease) when the new mocha is being used with third-party related modules (e.g. ts-mocha).

[ceisele-r]

@juergba thank you so much, this was it. I had to remove one --…

I don’t know why this suddenly had to be changed now as we did not change something other than updating mocha, but it’s working now again and I have something to keep in mind for the future.

Thanks again!

[ceisele-r]

@juergba thanks for the fast response.

Actually, the intention to use ts-mocha was to improve test start time (only transpile code without type checks instead of doing full type checks with tsc-preprocessing). I think even ts-mocha@next does not support Mocha v9 (see this comment).

I think we just missed that ts-mocha does not support Mocha v9, sorry for that. And actually, once I remove ts-mocha from the scripts and instead perform pre-processing using tsc and then simply execute the tests using mocha ....., there is no longer a performance difference between v9.0.3 and v9.1.0.

So I guess for some reason, our (incorrect) way of “starting” mocha with ts-mocha now (with mocha v9.1.0) caused the performance issue.

So definitely not a mocha issue but maybe someone else runs into this and finds this information useful.

Thanks for guiding me in the right direction!