Discovered tests may have problematic filenames.

The testing adapter returns the following data for discovery (one item per root):

[
    {
        "rootid": "<string>",
        "root": "<dirname>",
        "parents": [
            {
                "id": "<string>",
                "kind": "<enum-ish>",
                "name": "<string>",
                "parentid": "<string>"
            }
        ],
        "tests": [
            {
                "id": "<string>",
                "name": "<string>",
                "source": "<filename>:<lno>",
                "markers": ["<string>"],
                "parentid": "<string>"
            }
        ]
    }

For each root there is one property that holds a directory path (“root”, AKA the test root). For each test there is one property that contains a filename (“source”). In both cases the extension cares about how the value maps (respectively) to the current workspace folder (or under it) and to a file known to the extension under that workspace root. (We could support test roots under other workspace folders, in the multi-root case, or even outside the workspace. However, that’s outside the scope of this issue.)

The problem is that we have no guarantee that the test root will be under the current workspace root, nor that any test’s source file will be a file known to the extension. Furthermore, there is not guarantee that either will have the same casing as the filename known to VS Code, on case-insensitive file systems.

Most of the time this isn’t an issue. However, in the case of pytest, plugins can generate filenames that break our assumptions. On top of that, from what I can tell test roots outside the workspace are ignored and normcase issues only result in an extra editor window getting opened. So the consequences aren’t severe.

The solution for the test root is along the lines of what was merged for #6755 (and later reverted in #6780). In src/client/testing/common/services/discoveredTestParser.ts we try to match data.root to the workspace root (or one of them for multi-root), either exactly or as a subdirectory, ignoring case.

The solution for each test’s “source” is trickier since doing it in TestDiscoveredTestParser.parse() would require an MxN comparison between the tests and every file known to VS Code. And if VS Code doesn’t have an API to quickly identify files in the workspace (either a list or some sort of workspace.hasFile()) then we would have to walk the filesystem. Either way it would probably be too expensive. So the check likely has to be done at each of the sites where the test source filename is actually used. This is how it is done for code lenses in #6782 (to solve #6303). Extending that to all other similar sites, possibly using a new helper, would be necessary. The solution isn’t ideal since it any new consumers of the test data don’t automatically get the check, but it’s probably the best we can do.

[ericsnowcurrently]

@PhilMacKay, issues #6410 and #6695 should be fixed now in the insiders build of the extension. (Feel free to validate.) This issue is more about some corner cases that are unlikely to affect most users.

[luabud]

This seems not to be a problem for users, so closing for now.