Running tests with workers causing incorrect failure / passing count

What are you trying to achieve?

To run a series of test features expect 0 or positive failures, or 100% success across all 78 tests.

What do you get instead?

In the output below, one test failed but the output says all test have passed (although there are 78 total, the count says 77.)

[5]   ✔ [testname]
[2]   ✔ [testname]
[8]   ✖ [testname] 264128ms
[8] [featurename] --
[8]   ✔ [testname]
[5]   ✔ [testname]
[2]   ✔ [testname]
[2]   ✔ [testname]
[5]   ✔ [testname]
[5]   ✔ [testname]
[2]   ✔ [testname]
[5]   ✔ [testname]
[2]   ✔ [testname]
[5]   ✔ [testname]
[5]   ✔ [testname]
[5]   ✔ [testname]
[5]   ✔ [testname]
[5]   ✔ [testname]
[5]   ✔ [testname]

  OK  | 77 passed   // 1106.326s
/tmp/build/3e139e9d
+ '[' 0 -ne 0 ']'

In other instances, with no pipeline changes, it would say all tests passed with -1 failed.

  FAIL  | 78 passed, -1 failed   // 1144.407s

Codecept is being run with:

codeceptjs run-workers --suites 8 --grep @full-suite

These asynchronous tests were passing OK for some time. They are all being ran within a codecept container.

Details

• CodeceptJS version: v2.4.1
• NodeJS Version: v12.10.0
• Operating System: Not sure
• puppeteer || webdriverio || protractor || testcafe version (if related) Not sure
• Configuration file:

exports.config = {
  tests: './tests/*.js',
  timeout: 10000,
  output: './output',
  helpers: {
    Puppeteer: {
      chrome: { args: ['--no-sandbox'] },
      url: 'http://localhost:8000',
      show: false,
      restart: false,
      keepBrowserState: true,
      windowSize: '1200x800',
      waitForNavigation: 'networkidle0',
      waitForTimeout: 30000,
    },
    DomHelpers: { require: './helpers/DomHelpers.js' },
  },
  include: {
    I: './steps/steps_file.js',
    [more stuff]
  },
  plugins: {
    retryFailedStep: {
      enabled: true,
      retries: 5,
      minTimeout: 500,
    },
    [customplugin]
  },
  bootstrap: false,
  // bail will fail hard after one test failure
  mocha: { bail: false },
  name: '[name]',
};

Hopefully the info I’ve provided is sufficient – please let me know if I can provide anything else!

[KMKoushik]

@Brydom Thanks for raising the issue. And yeah! you are correct.

This issue will occur if Scenario is same for two tests. I will think of a way to fix it. Meanwhile, you can test this by changing Scenario for two tests and run test on those alone.

var assert = require('assert');
var tries = 0;

Feature('Retry in workers');

Scenario('Flaky scenario', { retries: 2 }, () => {
  setTimeout(() => { tries++ }, 200);
  assert.equal(tries, 1);
});

Scenario('Flaky scenario', { retries: 2 }, () => {
  setTimeout(() => { tries++ }, 200);
  assert.equal(tries, 2);
});

This failed for me.

var assert = require('assert');
var tries = 0;

Feature('Retry in workers');

Scenario('Flaky scenario', { retries: 2 }, () => {
  setTimeout(() => { tries++ }, 200);
  assert.equal(tries, 1);
});

Scenario('Flaky scenario 1', { retries: 2 }, () => {
  setTimeout(() => { tries++ }, 200);
  assert.equal(tries, 2);
});

But this passed correctly

[anil-devaki]

when I am running with workers my test cases under single tag its able to execute my test cases and are passed . But when I try to run with 2 tags with 2 workers then two browsers opening but only one browser action are performing and the other one is in idle state.

can any one help me on this?