Support for "related code" in tests

@connor4312 Is there any consideration to how I would present one or more tests as being related to a code item like a function?

For instance, in Pester you can tag tests with anything. If I allowed a special tag that let you say a test was assigned to a certain function named Invoke-MyFunction, I’d want to expose a “play” button right next to the Invoke-MyFunction definition to run all tests “tagged” to that function. In this case though I don’t want it to be tracked with a separate ID, I want it to invoke those individual tests as part of the tag “rollup” which might be scattered across different files. Also tricky is how you would present a failure to the user in this case.

Ideally this would work with AutoRun as well whenever I change the function.

At any rate this is definitely a “nice to have” as opposed to a “must have” and could come at a later iteration, just a thought that would make testing really simple for people and not have to lose the context of where they are directly working on the application. I was inspired by your markdown test adapter example.

With the existing implementation it could just be offered under a “Function Test Rollup” header or something and aggregate the results together somehow, but when run it wouldn’t update the “referenced” tests necessarily (though I guess I could do this within the context of the extension) so its fragmented and not ideal.

_Originally posted by @JustinGrote in https://github.com/microsoft/vscode/issues/107467#issuecomment-861177620_

[connor4312]

Thanks for the feedback, all.

From feedback, I think one approach we can take is, like we do on tests, something of a hybrid:

• Have relatedCode?: vscode.Location[] on the TestItem.
  • I do not want to go with the approach of using the DocumentSymbol as @firelizzard18 suggested, as this type is something that LSP/languages understands, but testing doesn’t really. We don’t generally share types across domains except for a few common types (like URI).
  • Having this as a standalone property allows test extensions to discover related code like they do tests today – lazily based on editor visibility, eagerly if they’re very quick, or on a stronger signal if they’re not so fast. This problem of serving both fast and expensive test discovery is quite reminiscent to tests themselves which resulted in the semi-lazy model we have today, so reusing this mechanism would make sense to me.
  • @firelizzard18 wants to be a user of this style of relations 🙂
• Have an optional handler so that extensions can lazily serve related code only when explicitly asked for
  • If discovery of related code cannot reasonably be done in realtime, allow a provider containing the proposed provideImplementations(testItem: vscode.TestItem): ProviderResult<vscode.Location[]> / provideAssociatedTests(location: vscode.Location): ProviderResult<vscode.TestItem[]> methods to be added to the TestController.
  • This is in addition to the relatedCode? property. These should return ranges and test items rather than adding to the relatedCode property, since the relations are point-in-time and should not be persisted and tracked like the TestItem property.
  • Feedback: do we need this handler – who’s interested in it? I dislike having two ways to do the same thing, but I think the former proposal would be unsuitable for any ‘slower’ test runners

cc @jdneo / @kimadeline / @JustinGrote

[jdneo]

On the Python side there is no way to automatically or easily infer which tests are related to a specific piece of code (no naming conventions), so we would probably not use this feature as it stands right now.

From a general standpoint, I do agree with the handler approach, this way in the extension we might come up with a few simple rules that may help with basic inference.

Same in Java, lots of Java developers tends to append Test(s) at the end of the class name. But that’s not part of the Java spec. The handler approach can let us do some ‘guess’ action to find the related test code.