Code Coverage Metrics

Hi, I would like to explore implementing test coverage metrics in our codebase. Ideally, we would have something that runs as part of our CI process that provides an overall test coverage metrics as well as the ability to drill down and see the contributing factors to those high level metrics. We can also have that information surfaced in some form in open PRs.

Here are what I see as the goals/benefits of that. Do these seem reasonable? Anything I’m missing that we’d like to call out?

Goals:

• Provide a baseline understanding of how thoroughly different parts of our codebase are tested.
• Help identify areas we’d like to test more thoroughly and if there are any areas that we feel are overly tested. This could result in tech-debt tasks to add coverage on certain areas, or allow us to mark tests we’d remove when they require upkeep.
• Ensure we are intentional about increasing or decreasing code coverage over time through the PR process.

att. @desktop/maintainers

[shiftkey]

👍 to this work.

Some team-based ideas off the top of my head:

• when updating a component, review whether it can be simplified by extracting functionality that makes it easier to test
• reviewers encouraging PRs for fixes to also include regression tests, where appropriate (e.g. #5434)

Some specific “nice to have” integration ideas:

• PR integration to indicate how an incoming PR impacts the code coverage stats
• tracking trends over time about how we’re keeping up with coverage goals

[shiftkey]

Just noticed that Appveyor doesn’t have gcov installed - which is part of GCC - and it quietly errors (but still seems to succeed uploading code coverage):

yarn run v1.10.1
$ codecov -f coverage/*.json
  _____          _  
 / ____|        | |  
| |     ___   __| | ___  ___ _____   __  
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /  
| |___| (_) | (_| |  __/ (_| (_) \ V /  
 \_____\___/ \__,_|\___|\___\___/ \_/  
                                v3.1.0
==> Detecting CI Provider
    AppVeyor CI Detected
==> Configuration: 
    Endpoint: https://codecov.io
{ commit: 'bd3938780b3977a824c0f17727fd49d5038aa6ac',
  branch: '__release-beta-bd39387-112930252',
  package: 'node-v3.1.0' }
==> Building file structure
==> Generating gcov reports (skip via --disable=gcov)
    $ for /f "delims=" %g in ('dir /a-d /b /s *.gcno ') do gcov  %g
File Not Found
    Failed to run gcov command.
==> Targeting specific file
    X Failed to read file at C:\projects\desktop\coverage\*.json
==> Uploading reports
    Success!
    View report at: https://codecov.io/github/desktop/desktop/commit/bd3938780b3977a824c0f17727fd49d5038aa6ac
Done in 74.76s.

It might be cool to try out whether this can be disabled because the command line yarn test:unit:cov generates these JSON files that match coverage/*.json:

$ ls coverage/*.json
coverage/coverage-final.json