Running `npm react-scripts test` ignores the `--coverage` option

Describe the bug

As per #7164, running npm react-scripts test ignores the --coverage option for jest - that is, jest will pass even if coverage thresholds are not reached. Note that this is almost certainly NOT an issue with jest - it is an issue with create-react-app’s configuration of jest.

Did you try recovering your dependencies?

I’ve tried this with both yarn version 1.16.0 and 1.17.3 (thanks for reminding me to update yarn) - and I also removed node_modules and yarn.lock and then re-installed, which didn’t change anything.

Which terms did you search for in User Guide?

I didn’t search the user guide at all, I searched open + closed issues in this repo, and found #7164.

Environment

Environment Info:

System: OS: macOS High Sierra 10.13.6 CPU: (8) x64 Intel® Core™ i7-4771 CPU @ 3.50GHz Binaries: Node: 8.11.0 - ~/.nvm/versions/node/v8.11.0/bin/node Yarn: 1.17.3 - /usr/local/bin/yarn npm: 5.6.0 - ~/.nvm/versions/node/v8.11.0/bin/npm Browsers: Chrome: 75.0.3770.142 Firefox: Not Found Safari: 12.1.1 npmPackages: react: ^16.8.6 => 16.8.6 react-dom: ^16.8.6 => 16.8.6 react-scripts: 3.0.1 => 3.0.1 npmGlobalPackages: create-react-app: Not Found

Steps to reproduce

See $7164 for a trivial reproduction of the issue.

Expected behavior

The --coverage flag should fail tests when the coverage threshold is not met.

Actual behavior

The --coverage flag does not fail tests when the coverage threshold is not met.

More detailed explanation

Our repo is somewhat complex - we’re using react on the front-end (tested with jest) and node on our back-end (tested with the same jest binary). Here are the two commands we run in package.json:

"test-frontend": "react-scripts test --coverage --watchAll=false"
"test-backend": "jest --config=amplify/jest.config.js --coverage --watchAll=false"

Here’s some relevant parts of package.json:

"jest": {
    "snapshotSerializers": [
      "enzyme-to-json/serializer"
    ],
    "collectCoverageFrom": [
      // Snipped
    ],
    "coverageThreshold": {
      "global": {
        "statements": 98.38,
        "branches": 98.38,
        "functions": 96.36,
        "lines": 98.58
      }
    }
  },

And here’s jest.config.js:

module.exports = {
  collectCoverageFrom: [
    // Snipped
  ],
  coverageThreshold: {
    global: {
      statements: 100,
      branches: 91.67,
      functions: 100,
      lines: 100
    }
  },
  roots: [
    'backend/function',
    'backend/api/reactkart/customTransformers'
  ],
  testEnvironment: 'node'
}

For the test-backend command, jest will fail if the coverage thresholds are not met. However, jest still passes for the test-frontend command if thresholds are not met. I’m fairly confident this means the issues lies with create-react-app and not jest, as both commands use the same jest binary and have fairly minimal configuration.

[zjullion]

CI=true does help, but --bail does not:

However, using CI=true removes all the coloured highlighting (not the end of the world, but unfortunate), and I don’t believe it’s documented anywhere that you must set CI=true in order to get coverage to work…

Furthermore, I’ve found something even worse - if you don’t set CI=true or --bail, and have failing tests, jest still returns a 0 exit code:

Again, this is the same jest binary that runs our back-end tests, and those work correctly. This is a very serious bug, and if people don’t know to set CI=true specifically (that is, even using the --ci option doesn’t work) this will mean CI builds will succeed with failing tests.

[zjullion]

Can confirm this is fixed in 3.1.0 - great work!