Code Coverage report doesn't reflect reality coverage of hits and misses - highlighting random lines

Logs and screenshots

• screenshot of line coverage report, hit and miss lines don’t make sense on the .vue files. Not sure what exactly is being missed or hit by the test running against the app. Just seems kinda randomly placed everywhere.

I’m not sure if I understand this debug step here below. I don’t know where the logs are output or if this variable is set before starting the client app or before running the cypress tests. my npm run dev command may be slightly different than what is expected in the README.

Please provide debug logs by running Cypress from the terminal with DEBUG=code-coverage environment variable set, see the Debugging section of the README file.

Versions

• What is this plugin’s version? - latest - v3.10.0
• If the plugin worked before in version X, but stopped after upgrading to version Y, please try the released versions between X and Y to see where the breaking change was. - n/a
• What is Cypress version? - latest - v10.6.0
• What is your operating system? MacOS Catalina 10.15.7
• What is the shell? - GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin19)
• What is the Node version? - v16.14.2
• What is the NPM version? - v8.5.0
• How do you instrument your application? Cypress does not instrument web application code, so you need to do it yourself.

• I instrument the vue app with my babel.config.js file displayed below

const plugins = [];
if (process.env.NODE_ENV === "development") {
  plugins.push([
    "babel-plugin-istanbul",
    {
      extension: [".ts", ".js", ".vue"]
    }
  ]);
}
module.exports = {
  presets: ["@vue/cli-plugin-babel/preset"],
  plugins
};

• When running tests, if you open the web application in regular browser, and open DevTools, do you see window.__coverage__ object? Can you paste a screenshot? - yes

• screenshot of coverage object in application window in dev mode

• Is there .nyc_output folder? Is there .nyc_output/out.json file. Is it empty? Can you paste at least part of it so we can see the keys and file paths? - folder and file is there, not empty

{
  "<REDACTED>/Nomad-Lang/client/src/main.ts": {
    "path": "<REDACTED>/Nomad-Lang/client/src/main.ts",
    "statementMap": {
      "0": {
        "start": {
          "line": 2,
          "column": 22
        },
        "end": {
          "line": 4,
          "column": 1
        }
      },
      "1": {
        "start": {
          "line": 3,
          "column": 4
        },
        "end": {
          "line": 3,
          "column": 62
        }
      },
      "2": {
        "start": {
          "line": 5,
          "column": 0
        },
        "end": {
          "line": 5,
          "column": 62
        }
      },

• Do you have any custom NYC settings in package.json (nyc object) or in other NYC config files - yes

"nyc": {
    "all": true,
    "include": [
      "src/components/*.vue",
      "src/router/index.ts",
      "src/store/*.ts",
      "src/utils/*.ts",
      "src/views/*.vue",
      "src/App.vue",
      "src/main.ts"
    ]
  },

• Do you run Cypress tests in a Docker container? - no

Describe the bug A clear and concise description of what the bug is.

• It seems for some reason the hit and miss highlighting is not accurate at all in the .vue files, I cant really tell what I need to cover what's being missed in the .vue files. It just seems like it highlights random places and shows IF ELSE branching in random places where there is no IF ELSE branch in the line coverage report files.

Link to the repo Bugs with a reproducible example, like an open source repo showing the bug, are the most likely to be resolved.

• repo

• https://github.com/Dj-Viking/Nomad-Lang

• steps to reproduce

git clone [e-mail]:Dj-Viking/Nomad-Lang.git && \
cd Nomad-Lang && \
git checkout task/88/update-cypress && \
npm install && \
cd client && \
npm run test:cy:ci

• when tests are finished feel free to killall node as the concurrent shell command running the client dev server and the headless cypress tests, will keep a background shell running of the client dev server even with a SIGTERM in the current foreground shell.
• open coverage report in browser from client/coverage/lcov-report/index.html (right-click and run in live server or just open the index.html file in your browser of choice)
• look at any .vue file that was covered by the cypress tests

Thanks for your time,

Anders a.k.a. DJViking

[jogelin]

@Scott-Fischer I am using typescript (Angular templates) instrumenting with babel-plugin-istanbul + webpack too

But I was explaining that, because of the fixSourcePaths function that is setting the absolute path, I had an issue when trying to merge reports.

For example, if you are generating 2 reports containing the same file, then the absolute path will be different and then they will be considered as 2 different files. Si mainly by removing the absolute path, the path to the files are equals and the merge works correctly.

But I also have this issue locally concerning the highlight of the coverage using webpack and babel-plugin-istanbul.

[Scott-Fischer]

Another interesting development I just found.

My initial discovery of this problem was occurring in a completely customized tool chain that was set up manually. So there’s a lot of moving pieces/configs and it was really tough to track down the exact source of this problem. e.g. typescript, babel, webpack etc etc.

However, we have a separate react frontend client that was created with CRA. It was initially just a JS app and we had code coverage through @cypress/instrument-cra + cypress-io/code-coverage which was working fine.

We then converted the app to typescript and started slowly migrating .js files to typescript. However, the coverage report would only list .js files and that’s likely because @cypress/instrument-cra isn’t instrumenting the typescript files. In order to add support for typescript, I removed the @cypress/instrument-cra module and modified the babel config via CRACO in order to manually add babel-plugin-istanbul.

This ultimately worked in generating the coverage for our TS files. But to my surprise, the exact same issue started to occur where the statement.start and statement.end values in .nyc_output/out.json no longer match the actual source code positions of the corresponding statements.

I tried various ways of generating source maps (through TSC, through babel, through webpack, inline, external files, etc) and did not have any success in resolving the issue.

So I’m wondering, is the problem in the cypress coverage plugin? Or is the issue actually in the underlying babel-plugin-istanbul module? Because when using @cypress/instrument-cra to instrument the JS code, cypress-io/code-coverage worked perfectly.