question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Test Coverage API

See original GitHub issue

Notes researching of existing formats:

Clover

  • Contains timestamp and total lines of code, files, classes, coverage numbers (see below)
  • Also can include complexity information
  • Lines are covered as XML simply <line num="24" count="5" type="stmt"/>
  • Conditionals are represented as <line num="12" count="0" type="cond" truecount="0" falsecount="3"/>. Either truecount/falsecount being 0 indicates incomplete branch coverage.

Clover uses these measurements to produce a Total Coverage Percentage for each class, file, package and for the project as a whole. The Total Coverage Percentage allows entities to be ranked in reports. The Total Coverage Percentage (TPC) is calculated as follows:

TPC = (BT + BF + SC + MC)/(2*B + S + M) * 100%

where

BT - branches that evaluated to "true" at least once
BF - branches that evaluated to "false" at least once
SC - statements covered
MC - methods entered

B - total number of branches
S - total number of statements
M - total number of methods

gcov

Format info on the bottom of: http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php

  • Organized per test, which are associated with one or more files. Indicates functions and how many times each function was called.
  • Lines have number of times they were hit, DA:<line number>,<execution count>[,<checksum>]
  • Has similar branch coverage, but has indexed branches instead of true/false. A branch with 0 taken is uncovered.
    • BRDA:<line number>,<block number>,<branch number>,<taken>, declared multiple times for line, one for each branch
  • No timestamp

cobertura

  • General coverage information/count
  • Number of times each line was hit.
  • Has conditions in a slightly less-strict way, <line number="21" hits="58" branch="true" condition-coverage="50% (1/2)"/>
  • Normal lines look like <line number="20" hits="5" branch="false"/>
  • Has timestamp and metadata
  • Has method call count (everything is organized in classes)

This results in the following API. The TestCoverageProvider is given as an optional provider on the managed TestRun object. This is similar to the SourceControl interface in vscode.d.ts. The provider is then pretty standard; it’ll only be examined when the run finishes, so all its data is static.

It has a method to get general coverage information for all files involved in the run, and a method to provide detailed coverage for a URI.

I refer to the atomic unit of coverage as “statement coverage”. The examined test formats only provide line-based coverage, but statement coverage is what these are actually trying convey approximate and most tooling (e.g. istanbul/nyc) is technically capable of per-statement rather than per-line coverage. Therefore, it’s called statement coverage, but it can be used as line coverage.

The API is physically large due to the nature of data being provided, but is conceptually pretty simple (at least compared to the Test Controller API!). I don’t like the method names on the TestCoverageProvider, but have not thought of better ones yet.

/** ... */
export interface TestRun<T = void> {
	/**
	 * Test coverage provider for this result. An extension can defer setting
	 * this until after a run is complete and coverage is available.
	 */
	coverageProvider?: TestCoverageProvider
	// ...
}

/**
 * Provides information about test coverage for a test result.
 * Methods on the provider will not be called until the test run is complete
 */
export interface TestCoverageProvider<T extends FileCoverage = FileCoverage> {
	/**
	 * Returns coverage information for all files involved in the test run.
	 * @param token A cancellation token.
	 * @return Coverage metadata for all files involved in the test.
	 */
	provideFileCoverage(token: CancellationToken): ProviderResult<T[]>;

	/**
	 * Give a FileCoverage to fill in more data, namely {@link FileCoverage.detailedCoverage}.
	 * The editor will only resolve a FileCoverage once, and onyl if detailedCoverage
	 * is undefined.
	 * 
	 * @param coverage A coverage object obtained from {@link provideFileCoverage}
	 * @param token A cancellation token.
	 * @return The resolved file coverage, or a thenable that resolves to one. It
	 * is OK to return the given `coverage`. When no result is returned, the
	 * given `coverage` will be used.
	 */
	resolveFileCoverage(coverage: T, token: CancellationToken): ProviderResult<T>;
}

/**
 * A class that contains information about a covered resource. A count can
 * be give for lines, branches, and functions in a file.
 */
export class CoveredCount {
	/**
	 * Number of items covered in the file.
	 */
	readonly covered: number;
	/**
	 * Total number of covered items in the file.
	 */
	readonly total: number;

	/**
	 * @param covered Value for {@link CovereredCount.covered}
	 * @param total Value for {@link CovereredCount.total}
	 */
	constructor(covered: number, total: number);
}

/**
 * Contains coverage metadata for a file.
 */
export class FileCoverage {
	/**
	 * File URI.
	 */
	readonly uri: Uri;

	/**
	 * Statement coverage information. If the reporter does not provide statement
	 * coverage information, this can instead be used to represent line coverage.
	 */
	readonly statementCoverage: CoveredCount;
	
	/**
	 * Branch coverage information.
	 */
	readonly branchCoverage?: CoveredCount;
	
	/**
	 * Function coverage information.
	 */
	readonly functionCoverage?: CoveredCount;

	/**
	 * Detailed, per-statement coverage. If this is undefined, the editor will
	 * call {@link TestCoverageProvider.resolveFileCoverage} when necessary.
	 */
	detailedCoverage?: DetailedCoverage[];

	/**
	 * @param uri Covered file URI
	 * @param statementCoverage Statement coverage information. If the reporter
	 * does not provide statement coverage information, this can instead be
	 * used to represent line coverage.
	 * @param branchCoverage Branch coverage information
	 * @param functionCoverage Function coverage information
	 */
	constructor(
		uri: Uri,
		statementCoverage: CoveredCount,
		branchCoverage?: CoveredCount,
		functionCoverage?: CoveredCount,
	);
}

/**
 * Contains coverage information for a single statement or line.
 */
export class StatementCoverage {
	/**
	 * The number of times this statement was executed. If zero, the
	 * statement will be marked as un-covered.
	 */
	readonly executionCount: number;

	/**
	 * Statement position.
	 */
	readonly position: Position | Range;

	/**
	 * Coverage from branches of this line or statement. If it's not a
	 * conditional, this will be empty.
	 */
	readonly branches: readonly BranchCoverage[];

	/**
	 * @param location The statement position.
	 * @param executionCount The number of times this statement was
	 * executed. If zero,  the statement will be marked as un-covered.
	 * @param branches Coverage from branches of this line.  If it's not a
	 * conditional, this should be omitted.
	 */
	constructor(executionCount: number, location: Position | Range, branches?: BranchCoverage[]);
}

/**
 * Contains coverage information for a branch of a {@link StatementCoverage}.
 */
export class CoverageBranch {
	/**
	 * The number of times this branch was executed. If zero, the
	 * branch will be marked as un-covered.
	 */
	readonly executionCount: number;

	/**
	 * Branch position.
	 */
	readonly position?: Position | Range;

	/**
	 * @param executionCount The number of times this branch was executed.
	 * @param location The branch position.
	 */
	constructor(executionCount: number, location?: Position | Range);
}

/**
 * Contains coverage information for a function or method.
 */
export class FunctionCoverage {
	/**
	 * The number of times this function was executed. If zero, the
	 * function will be marked as un-covered.
	 */
	readonly executionCount: number;

	/**
	 * Function position.
	 */
	readonly position: Position | Range;

	/**
	 * @param executionCount The number of times this function was executed.
	 * @param location The function position.
	 */
	constructor(executionCount: number, location: Position | Range);
}

export type DetailedCoverage = StatementCoverage | FunctionCoverage;

Issue Analytics

  • State:open
  • Created 2 years ago
  • Reactions:11
  • Comments:22 (20 by maintainers)

github_iconTop GitHub Comments

6reactions
connor4312commented, Jun 20, 2022

It’s coming, but still a couple iterations out; my time is currently used by a new thing that’ll be announced soon.

3reactions
gjsjohnmurraycommented, Feb 4, 2022

I implement the coverage API and data flow for it internally. There’s no UI for it, but it’s there to try out. At some point in the next few weeks UI will magically show up ✨

@connor4312 what is the current status of UI for the coverage API? I’m not seeing any when I run test-provider-sample even though it seems to implement coverage.

Read more comments on GitHub >

github_iconTop Results From Across the Web

5 Introduction to API Coverage Tool
The API Coverage tool is used to estimate the test coverage that a test suite under development affords to implementations of its related...
Read more >
Test Design and Coverage for a REST API | by Venkata Sai
API Testing Approach is a predefined strategy or a method that the QA team will perform. This helps to better understand the functionalities, ......
Read more >
REST Coverage Testing | ReadyAPI Documentation
The Coverage page displays the operations defined in your API specifications and the hierarchy of test suites and test cases you have in...
Read more >
API Test Coverage - SoapUI
Quickly analyze API test coverage in ReadyAPI ... ReadyAPI can show you exactly where to focus by measuring the test coverage of your...
Read more >
Tests You Should Add To Your API Test Coverage To Become ...
Tests You Should Add To Your API Test Coverage To Become A Legend! Within this guide, you will get access to a checklist....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found