Test Coverage API

Notes researching of existing formats:

Clover

• Contains timestamp and total lines of code, files, classes, coverage numbers (see below)
• Also can include complexity information
• Lines are covered as XML simply <line num="24" count="5" type="stmt"/>
• Conditionals are represented as <line num="12" count="0" type="cond" truecount="0" falsecount="3"/>. Either truecount/falsecount being 0 indicates incomplete branch coverage.

Clover uses these measurements to produce a Total Coverage Percentage for each class, file, package and for the project as a whole. The Total Coverage Percentage allows entities to be ranked in reports. The Total Coverage Percentage (TPC) is calculated as follows:

TPC = (BT + BF + SC + MC)/(2*B + S + M) * 100%

where

BT - branches that evaluated to "true" at least once
BF - branches that evaluated to "false" at least once
SC - statements covered
MC - methods entered

B - total number of branches
S - total number of statements
M - total number of methods

gcov

Format info on the bottom of: http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php

• Organized per test, which are associated with one or more files. Indicates functions and how many times each function was called.
• Lines have number of times they were hit, DA:<line number>,<execution count>[,<checksum>]
• Has similar branch coverage, but has indexed branches instead of true/false. A branch with 0 taken is uncovered.
  • BRDA:<line number>,<block number>,<branch number>,<taken>, declared multiple times for line, one for each branch
• No timestamp

cobertura

• General coverage information/count
• Number of times each line was hit.
• Has conditions in a slightly less-strict way, <line number="21" hits="58" branch="true" condition-coverage="50% (1/2)"/>
• Normal lines look like <line number="20" hits="5" branch="false"/>
• Has timestamp and metadata
• Has method call count (everything is organized in classes)

This results in the following API. The TestCoverageProvider is given as an optional provider on the managed TestRun object. This is similar to the SourceControl interface in vscode.d.ts. The provider is then pretty standard; it’ll only be examined when the run finishes, so all its data is static.

It has a method to get general coverage information for all files involved in the run, and a method to provide detailed coverage for a URI.

I refer to the atomic unit of coverage as “statement coverage”. The examined test formats only provide line-based coverage, but statement coverage is what these are actually trying convey approximate and most tooling (e.g. istanbul/nyc) is technically capable of per-statement rather than per-line coverage. Therefore, it’s called statement coverage, but it can be used as line coverage.

The API is physically large due to the nature of data being provided, but is conceptually pretty simple (at least compared to the Test Controller API!). I don’t like the method names on the TestCoverageProvider, but have not thought of better ones yet.

/** ... */
export interface TestRun<T = void> {
	/**
	 * Test coverage provider for this result. An extension can defer setting
	 * this until after a run is complete and coverage is available.
	 */
	coverageProvider?: TestCoverageProvider
	// ...
}

/**
 * Provides information about test coverage for a test result.
 * Methods on the provider will not be called until the test run is complete
 */
export interface TestCoverageProvider<T extends FileCoverage = FileCoverage> {
	/**
	 * Returns coverage information for all files involved in the test run.
	 * @param token A cancellation token.
	 * @return Coverage metadata for all files involved in the test.
	 */
	provideFileCoverage(token: CancellationToken): ProviderResult<T[]>;

	/**
	 * Give a FileCoverage to fill in more data, namely {@link FileCoverage.detailedCoverage}.
	 * The editor will only resolve a FileCoverage once, and onyl if detailedCoverage
	 * is undefined.
	 * 
	 * @param coverage A coverage object obtained from {@link provideFileCoverage}
	 * @param token A cancellation token.
	 * @return The resolved file coverage, or a thenable that resolves to one. It
	 * is OK to return the given `coverage`. When no result is returned, the
	 * given `coverage` will be used.
	 */
	resolveFileCoverage(coverage: T, token: CancellationToken): ProviderResult<T>;
}

/**
 * A class that contains information about a covered resource. A count can
 * be give for lines, branches, and functions in a file.
 */
export class CoveredCount {
	/**
	 * Number of items covered in the file.
	 */
	readonly covered: number;
	/**
	 * Total number of covered items in the file.
	 */
	readonly total: number;

	/**
	 * @param covered Value for {@link CovereredCount.covered}
	 * @param total Value for {@link CovereredCount.total}
	 */
	constructor(covered: number, total: number);
}

/**
 * Contains coverage metadata for a file.
 */
export class FileCoverage {
	/**
	 * File URI.
	 */
	readonly uri: Uri;

	/**
	 * Statement coverage information. If the reporter does not provide statement
	 * coverage information, this can instead be used to represent line coverage.
	 */
	readonly statementCoverage: CoveredCount;
	
	/**
	 * Branch coverage information.
	 */
	readonly branchCoverage?: CoveredCount;
	
	/**
	 * Function coverage information.
	 */
	readonly functionCoverage?: CoveredCount;

	/**
	 * Detailed, per-statement coverage. If this is undefined, the editor will
	 * call {@link TestCoverageProvider.resolveFileCoverage} when necessary.
	 */
	detailedCoverage?: DetailedCoverage[];

	/**
	 * @param uri Covered file URI
	 * @param statementCoverage Statement coverage information. If the reporter
	 * does not provide statement coverage information, this can instead be
	 * used to represent line coverage.
	 * @param branchCoverage Branch coverage information
	 * @param functionCoverage Function coverage information
	 */
	constructor(
		uri: Uri,
		statementCoverage: CoveredCount,
		branchCoverage?: CoveredCount,
		functionCoverage?: CoveredCount,
	);
}

/**
 * Contains coverage information for a single statement or line.
 */
export class StatementCoverage {
	/**
	 * The number of times this statement was executed. If zero, the
	 * statement will be marked as un-covered.
	 */
	readonly executionCount: number;

	/**
	 * Statement position.
	 */
	readonly position: Position | Range;

	/**
	 * Coverage from branches of this line or statement. If it's not a
	 * conditional, this will be empty.
	 */
	readonly branches: readonly BranchCoverage[];

	/**
	 * @param location The statement position.
	 * @param executionCount The number of times this statement was
	 * executed. If zero,  the statement will be marked as un-covered.
	 * @param branches Coverage from branches of this line.  If it's not a
	 * conditional, this should be omitted.
	 */
	constructor(executionCount: number, location: Position | Range, branches?: BranchCoverage[]);
}

/**
 * Contains coverage information for a branch of a {@link StatementCoverage}.
 */
export class CoverageBranch {
	/**
	 * The number of times this branch was executed. If zero, the
	 * branch will be marked as un-covered.
	 */
	readonly executionCount: number;

	/**
	 * Branch position.
	 */
	readonly position?: Position | Range;

	/**
	 * @param executionCount The number of times this branch was executed.
	 * @param location The branch position.
	 */
	constructor(executionCount: number, location?: Position | Range);
}

/**
 * Contains coverage information for a function or method.
 */
export class FunctionCoverage {
	/**
	 * The number of times this function was executed. If zero, the
	 * function will be marked as un-covered.
	 */
	readonly executionCount: number;

	/**
	 * Function position.
	 */
	readonly position: Position | Range;

	/**
	 * @param executionCount The number of times this function was executed.
	 * @param location The function position.
	 */
	constructor(executionCount: number, location: Position | Range);
}

export type DetailedCoverage = StatementCoverage | FunctionCoverage;

[connor4312]

It’s coming, but still a couple iterations out; my time is currently used by a new thing that’ll be announced soon.

[gjsjohnmurray]

I implement the coverage API and data flow for it internally. There’s no UI for it, but it’s there to try out. At some point in the next few weeks UI will magically show up ✨

@connor4312 what is the current status of UI for the coverage API? I’m not seeing any when I run test-provider-sample even though it seems to implement coverage.