RemoteExecutor Swallows Errors and Throws Invalid Exceptions

• This issue is blocking
• This issue is causing unreasonable pain

I could be holding it wrong but it does not seem to be working as expected.

Microsoft.DotNet.RemoteExecutor Version 6.0.0-beta.20508.3 (Are there non beta version available?)

Installed via https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json

The following method runs without any assertation exceptions. I would expect it to fail.

            RemoteExecutor.Invoke(() =>
            {
                 Assert.True(false);
            }).Dispose();

The following code throws an exception. I would expect none.

            RemoteExecutor.Invoke(() =>
            {
                 return RemoteExecutor.SuccessExitCode;
            }).Dispose();

Message: Exit code was -2147450749 but it should have been 42 Expected: True Actual: False Stack Trace: RemoteInvokeHandle.Dispose(Boolean disposing) line 237 RemoteInvokeHandle.Dispose() line 58

[ViktorHofer]

Thanks for reporting @JimBobSquarePants and @vitek-karas for the analysis. Vitek is right that RemoteExecutor doesn’t set HostRunner (the path to the host to invoke) to the right value in cases where VSTest’s apphost is used. This didn’t show up for us in dotnet/runtime as we don’t use VSTest’s apphost feature as we supply our custom host.

As a simple fallback mechanism we could check if processFileName ends with (dotnet[.exe]) and if not, use the dotnet in the %PATH%.

For a more complete implementation, we should follow what Vitek proposes:

It should be using dotnet.exe - but this will be tricky because it will need to pick the right one. The best way would probably be to use location of the runtime running the test (this will be tricky to get completely right with single-file, but it’s possible).

An implementation of that could look like this:

1. Retrieve the location of the shared framework assemblies currently being used by the process.
2. Climb up the directory tree until a dotnet host is found.
3. If none is found, use the one in the PATH.

@vitek-karas do you think we should stop at some point climbing up the directory tree? For self-contained application we need to fallback to %PATH% as a self-contained application doesn’t have a dotnet host. Right?

[vitek-karas]

Thanks - I was able to repro it locally.

The command line it executes doesn’t seem to make sense to me:

F:\AppModel\repro\RemoteExecutorTests\ExecutorTest\bin\Debug\netcoreapp3.1\testhost.exe 
exec 
--runtimeconfig "F:\AppModel\repro\RemoteExecutorTests\ExecutorTest\bin\Debug\netcoreapp3.1\ExecutorTest.runtimeconfig.json" 
--depsfile "F:\AppModel\repro\RemoteExecutorTests\ExecutorTest\bin\Debug\netcoreapp3.1\ExecutorTest.deps.json" 
"F:\AppModel\repro\RemoteExecutorTests\ExecutorTest\bin\Debug\netcoreapp3.1\Microsoft.DotNet.RemoteExecutor.dll"
<args to the remote executor - not interesting>

This would work if it were running “dotnet.exe exec …”, but I doubt testhost recognizes “exec” (I could be wrong though).

In any case the command line won’t work because the special host parameters must be first on the command line: --runtimeconfig and so on. Since testhost.exe is an apphost it doesn’t inherently recognize exec on the host level, and so it runs itself as if there were no special parameters on the command line. And thus it tries to read the file testhost.runtimeconfig.json which doesn’t exist (the valid=[1] is misleading, internally the host treats missing runtime config as valid) - so it ends up with empty runtime config which means no framework references and thus tries to run as self-contained - which it’s not and so on.

I tried running the same command line via dotnet.exe exec and that works (well at least it runs the code).

I don’t know how this passes tests (if they run) in Arcade repo and/or how it’s used in a case where it would actually work. But as is, this seems completely broken.

Also - I which we could update the testhost.exe to some “recent” version, it’s 2.1 (I know we probably need it to be able to run 2.1 tests, unfortunately). In this case it means that the tracing from the testhost.exe is not visible in the trace file since it doesn’t recognize COREHOST_TRACEFILE which was added in 3.0.

I think somebody from Arcade should comment on what is the intended command line. I can definitely help with figuring out what it should be in reality to get all the desired behaviors.

Couple of notes about what I think it SHOULD be doing:

• Using testhost.exe won’t work because that is the VS test host runner, it does not support running the executor dll as a command line app
• It should be using dotnet.exe - but this will be tricky because it will need to pick the right one. The best way would probably be to use location of the runtime running the test (this will be tricky to get completely right with single-file, but it’s possible).

The break was probably introduced by changes to the VSTest platform - some time ago it switched from running the test via “dotnet.exe” to using the “testhost.exe”. But that’s been quite a while now if I remember correctly.