Merging coverage files doesn't produce accurate cumulative result

I execute the following nyc commands, after generating individual coverage reports on the same codebase using different test executions run by Jest and Cypress.

nyc merge coverage-jest combined/coverage-jest.json nyc merge coverage-cypress combined/coverage-cypress.json nyc report combined --reporter html --reporter text

Expected Behavior

I’d expect this to show a correct cumulative coverage report of .ts and .html files.

Observed Behavior

1. Coverage of .html files is not shown in merged report, while they are show in individial report.
2. Coverage of some .ts files is shown correctly, but for others a Cannot read property 'locations' of undefined TypeError is shown.

Troubleshooting steps

• still occurring when I put cache: false in my nyc config

Environment Information

  System:
    OS: Windows 10 10.0.18363
    CPU: (8) x64 Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
    Memory: 7.97 GB / 31.88 GB
  Binaries:
    Node: 13.6.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.0 - C:\Program Files\nodejs\yarn.CMD
    npm: 6.13.4 - C:\Program Files\nodejs\npm.CMD
  npmPackages:
    istanbul-instrumenter-loader: 1.2.0 => 1.2.0
    istanbul-lib-coverage: 1.2.1 => 1.2.1
    nyc: ^15.0.1 => 15.0.1
    source-map-support: ^0.5.16 => 0.5.16
    ts-node: ^7.0.1 => 7.0.1
    typescript: ~3.7.4 => 3.7.5

For Cypress coverage I use above dependencies as well as @cypress/code-coverage:^3.0.2. I’m using jest: 24.1.0 to produce unit test coverage.

[CharlesStover]

I’m experiencing this same problem.

My coverage results look like so:

cypress/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "1": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": 3 }
      },
      "2": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "3": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "4": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "5": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "6": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 47 },
          "end": { "line": 9, "column": 1 }
        },
        "line": 3
      },
      "1": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": 3 }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": 3 }
          },
          { "start": {}, "end": {} }
        ],
        "line": 5
      },
      "1": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 6, "1": 4, "2": 2, "3": 4, "4": 2, "5": 2, "6": 2 },
    "f": { "0": 4, "1": 2 },
    "b": { "0": [0, 4], "1": [2] },
    "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
    "hash": "81128de9166cac88dff871fa724f973dcd80167a"
  }
}

jest/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "1": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "2": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "3": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "4": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "5": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 1, "1": 1, "2": 1, "3": 1, "4": 0, "5": 1 },
    "f": { "0": 1 },
    "b": { "0": [1] }
  }
}

The expected outcome would be path/to/file.ts having 100% coverage, since the gaps on one test are covered by the other test. However, instead, what you see is the gaps being reported:

combined/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "1": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": 3 }
      },
      "2": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "3": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "4": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "5": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "6": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 47 },
          "end": { "line": 9, "column": 1 }
        },
        "line": 3
      },
      "1": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": 3 }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": 3 }
          },
          { "start": {}, "end": {} }
        ],
        "line": 5
      },
      "1": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 7, "1": 4, "2": 3, "3": 4, "4": 3, "5": 3, "6": 3 },
    "f": { "0": 4, "1": 3 },
    "b": { "0": [0, 4], "1": [3] },
    "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
    "hash": "81128de9166cac88dff871fa724f973dcd80167a"
  }
}

The “coverage” merge is really just concatenated instead of merging.

It appears that: line 1 characters 1-10 uncovered + line 1 characters 1-5 covered + line 1 characters 5-10 covered = line 1 characters 1-10 uncovered (wrong) + line 1 characters 1-10 covered (right).

The uncovered characters are still shown as uncovered even though their character-span is covered by a different part of the map. Just guessing.

Here is a nice visual LCOV report for the above JSON:

Cypress (fully covered except for 1 branch):

Jest (specifically covering the above branch):

Combined report (expected to be fully covered):

Any advice to unblock this? I don’t want to have to write a JSON merger manually. 😓

Maybe this is just a weird issue with TypeScript.

Below, how are imports and exports not covered? The export statement isn’t covered, but the inside of the function somehow is. The inside references an import, but the import isn’t covered. Nonsensical. 😵

[jogelin]

@D0rmouse did you finally find a good configuration to combine multiple coverage reports?

I have the same type of stack with Nx + Cypress + Storybook + Jest, generating reports separately works fine but when I want to merge, it is not good.

It makes sense because each tool are using different versions for each generation.