Coverage reporting in codecoverage.json and test-result.txt files is sometimes inconsistent

Summary

Apex Code Coverage by Class as reported in the test-result.txt file is not consistent with code coverage details reported in the codecoverage.json file for some classes.

Steps To Reproduce:

Repository to reproduce: dreamhouse-lwc

This may be difficult to reproduce against dummy data. Class reporting is only inaccurate for some files, and the inaccuracies only show up when run against a custom coverage parsing tool (see bottom for explanation) that uncovers the inconsistencies between the files. I need to run my parsing tool against dreamhouse-lwc to get more data and will pin any resulting screenshots here - but wanted to open the issue while it’s fresh in my mind.

To reproduce, you may either have to create your own coverage parsing function that emulates mine, or step through the codecoverage.json file manually and compare the results with the test-results.txt file.

Expected result

A single class of the Code Coverage By Class section of the test-result.txt file should reflect the aggregated coverage for that class as reported in the codecoverage.json file.

Actual result

The Code Coverage By Class section is inconsistent with the coverage reported by in the codecoverage.json file and reports higher coverage than is accurate for some classes.

Example: In the test-result.txt file, the Code Coverage By Class section reports the ExampleClass file as having 66.7% coverage.

The Apex Code Coverage for Test Run {ID} section of the same file reports that ExampleClass was a Class Being Tested 31 times. This is consistent with the codecoverage.json file, which also shows 31 instances of the ExampleClass as the class under test, with the same methods and percentages as shown in the breakdown of the Apex Code Coverage for Test Run {ID} section.

Expectation: Stepping through each occurrence of the 31 occurrences ExampleClass in codecoverage.json and building an array of covered/uncovered lines should ultimately reflect a coverage percentage that is consistent with what is reported in the Code Coverage By Class section.

Resulting coverage arrays from stepping through each instance of ExampleClass in codecoverage.json manually (removing duplicates): coveredLines: [16,17,18,19,20,21,22,34,35,36,37,38,39,40,42,43,45] uncoveredLines: [3,4,5,6,7,8,9,10,11,12,13,26,27,28,29,31]

Resulting number of covered lines: 17 Total number of lines that count towards coverage: 33

Coverage as reflected by aggregated output of codecoverage.json: 17/33 = 51.52% Coverage reported in Code Coverage By Class section: 66.7% (22/33)

The reported class coverage suggests 22 lines were covered instead of 17. Uncovered lines array is truncated but shows [3, 4, 5, 6, 7…] so I am assuming that lines 26-31 (representing one of two untested methods) are not appearing in the array of uncovered lines (removing these 5 lines from the uncovered array and adding them to the covered array adds up to 22/33 lines covered = 66.7%).

If these supposed ‘covered lines’ are not shown as covered anywhere in this class in the codecoverage.json file, where are they coming from?

System Information

sfdx version: 7.132.0

Command: sfdx force:apex:test:run
-d reports/apex
–codecoverage
–detailedcoverage
–testlevel=RunLocalTests
–resultformat=human
–targetusername=“${SCRATCH_TEST_ORG}”

Tests run on CircleCI against a deployed scratch org.

Additional information

I ran into this issue when writing a Javascript script to parse the codecoverage.json output provided by SFDX into a format that is readable by Coveralls. Once it was running, I noticed that the output in Coveralls was inconsistent with the reported “org-wide coverage” reported in the test-result.text file, and further, that running the tests against the same code produced inconsistent coverage results in the codecoverage.json file.

Steps for the script:

1. As the codecoverage.json file groups coverage output by method, aggregate coverage from each method into coverage by class.

• For each item in the coverage array, check if the apexClassOrTriggerId associated with this apexClassOrTriggerName already appears in the class map we’re building.
• If not, add the class with its covered/uncovered lines.
• If so, spread the covered and uncovered lines into the existing covered/uncovered lines array, removing duplicates and any newly covered lines from the uncovered array.

1. Build an array of source files in coveralls format, i.e. [null, 0, null, 1, 6, 99, null], where null represents lines that don’t count towards coverage (comments, whitespace), 0 represents no coverage, and positive integers represent the number of hits for that line.

• Using the map of class coverage, look for a file with a matching name. If a matching file is found, read the file contents and create an MD5 hash of the source code. Step through the array of covered/uncovered lines and build a new array in Coveralls source file format.

1. Post the coverage data to Coveralls and store both sfdx output and source_file output from parsing function as artifacts for reference.

Though my coverage parsing function is unit tested, I assumed I might have missed some edge cases, so I stepped through manually and compared the reporting by hand for each of these steps but I was unable to find errors. I’m pretty confident that what is shown in Coveralls reflects the details of the codecoverage.json file.

The ultimate question is: which file is accurate, the codecoverage.json file or the test-result.txt file? And why are they not consistent with each other?

Side note: if the codecoverage.json file is accurate, why might running tests against the same code produce different coverage results? Is this a likely consequence of running tests against a scratch org, or should I speak to the Salesforce developers about improving their unit testing / fixtures?

I’ve banged my head against this for a few weeks now - thanks in advance for any insight.

[madelineurl]

Thank you for the assistance @iowillhoit !

[iowillhoit]

Thank you for this incredibly detailed issue @madelineurl! This code is owned by another team, I will get it transferred over to their repo and I’ll post back to make sure GitHub alerts you of the new location.