Multi-proc build of a .sln file is 'The process cannot access the file because it is being used by another process.'

In the CLI repo, we have a Test.sln file which we dotnet build. However, we are seeing random failures that look like the following:

05:25:35.797     1>CSC : error CS2012: Cannot open 'D:\j\workspace\debug_windows---54dcb9ec\test\Msbuild.Tests.Utilities\obj\Debug\netcoreapp1.0\Msbuild.Tests.Utilities.dll' for writing -- 'The process cannot access the file 'D:\j\workspace\debug_windows---54dcb9ec\test\Msbuild.Tests.Utilities\obj\Debug\netcoreapp1.0\Msbuild.Tests.Utilities.dll' because it is being used by another process.' [D:\j\workspace\debug_windows---54dcb9ec\test\Msbuild.Tests.Utilities\Msbuild.Tests.Utilities.csproj] [D:\j\workspace\debug_windows---54dcb9ec\build.proj]

See https://ci.dot.net/job/dotnet_cli/job/rel_1.0.0/job/debug_windows_nt_x86_prtest/810/console

To repro:

1. On Windows x64 machine, clone the CLI repo and checkout https://github.com/dotnet/cli/pull/5465/commits/f6b0196b0f8a7342d4b5fe5709568af9e3ede665.
2. Then remove the line https://github.com/dotnet/cli/pull/5465/commits/f6b0196b0f8a7342d4b5fe5709568af9e3ede665#diff-70eb062c7648e65aa7aff25e924d014bR84 which is working around this problem by turning off multi-proc build.
3. build.cmd

An easier repro once you have built one time is:

1. F:\cli>git clean test -xdf
2. F:\cli>.dotnet_stage0\x64\dotnet.exe restore test\Microsoft.DotNet.Cli.Tests.sln
3. F:\cli>.dotnet_stage0\x64\dotnet.exe build test\Microsoft.DotNet.Cli.Tests.sln

F:\cli>.dotnet_stage0\x64\dotnet.exe build test\Microsoft.DotNet.Cli.Tests.sln
Microsoft (R) Build Engine version 15.1.523.56541
Copyright (C) Microsoft Corporation. All rights reserved.

F:\cli\.dotnet_stage0\x64\sdk\2.0.0-alpha-004616\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.Sdk.DefaultItems.targets(101,5): warning : A PackageReference for 'NETStandard.Library' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\Microsoft.DotNet.Tools.Tests.Utilities.csproj]
F:\cli\.dotnet_stage0\x64\sdk\2.0.0-alpha-004616\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.Sdk.DefaultItems.targets(101,5): warning : A PackageReference for 'NETStandard.Library' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\Microsoft.DotNet.Tools.Tests.Utilities.csproj]
  ArgumentsReflector -> F:\cli\test\ArgumentsReflector\bin\Debug\netcoreapp2.0\ArgumentsReflector.dll
  Microsoft.DotNet.Cli.Utils -> F:\cli\src\Microsoft.DotNet.Cli.Utils\bin\Debug\netstandard1.5\Microsoft.DotNet.Cli.Utils.dll
  Microsoft.DotNet.TestFramework -> F:\cli\src\Microsoft.DotNet.TestFramework\bin\Debug\netstandard1.5\Microsoft.DotNet.TestFramework.dll
  Microsoft.DotNet.InternalAbstractions -> F:\cli\src\Microsoft.DotNet.InternalAbstractions\bin\Debug\netstandard1.3\Microsoft.DotNet.InternalAbstractions.dll
CSC : error CS2012: Cannot open 'F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll' for writing -- 'The process cannot access the file 'F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll' because it is being used by another process.' [F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\Microsoft.DotNet.Tools.Tests.Utilities.csproj]

[rainersigwald]

That’s a great repro, @AlgorithmsAreCool, thanks!

C:\src\AlgorithmsAreCool-sdk-739>dotnet publish -r win7-x86 /logger:C:\src\ParallelBuildDebuggingLogger\ParallelBuildDebuggingLogger\bin\Debug\netstandard1.4\ParallelBuildDebuggingLogger.dll;CoreCompile /m:1 | findstr /rc:"Building target.*HigherLib.csproj"
Building target 'CoreCompile' in {4: "C:\src\AlgorithmsAreCool-sdk-739\HigherLib\HigherLib.csproj" + <SolutionDir = C:\src\AlgorithmsAreCool-sdk-739\; BuildingSolutionFile = true; Configuration = Debug; Platform = AnyCPU; SolutionFileName = AlgorithmsAreCool-sdk-739.sln; CurrentSolutionConfigurationContents = <SolutionConfiguration>
Building target 'CoreCompile' in {10: "C:\src\AlgorithmsAreCool-sdk-739\HigherLib\HigherLib.csproj" + <TargetFramework = netstandard1.6>}

This shows that the race is between

• The solution-driven invocation of HigherLib
  • New set of global properties is common to all builds invoked in the solution and contains solution metadata
  • Because this is a single-TFM project, this implicitly is the “inner” build, which does I/O work
• The ProjectReference-driven invocation of HigherLib for TargetFramework = netstandard1.6
  • New set of global properties is the explicit TFM desired for compatibility with App
  • Explicit request for the inner build

That explains a few questions I had (“why only in solutions?” and “why can’t I repro this with my multitargeting attempts?”).

@nguerrera @natidea I think we can fix this without going all the way to fixing https://github.com/Microsoft/msbuild/issues/1276 (which would probably be ideal but is invasive). Would it be reasonable to respond to the GetTargetFrameworkProperties request with no information if the project has a single TFM (just as we do for legacy projects)? That would collapse the latter request into the former and the engine would just return the previously-built file.

[rainersigwald]

No, the rule is that the engine builds only once for each tuple of:

• Project
• Target
• Set of global properties

It’s definitely concerning that something in the build is building project 4 22 times.

I’m going to apply some of my magic internal tracing tools to the Sdk repo to help chase this down.