Build fails due to locked file which it locks itself (concurrent build problem)

We moved from .net core 2.0 to 2.1. In this move, we retargeted several projects which were from netstandard2.0 to multitarget netcoreapp2.1 and netstandard2.0. Now we sometimes experience failed builds, both on CI servers and dev machines.

Q: I’d like to provide as much info as I can, so I could try collecting diag build logs and see if I can get one from failed build, would that help?

Steps to reproduce

No stable repro. Start a build using: dotnet publish -c Release -r win-x64 -f netcoreapp2.1

Expected behavior

Build works

Actual behavior

Sometimes the build stops with something like this:

build 11-Jul-2018 22:23:55 C:\Program Files\dotnet\sdk\2.1.301\Microsoft.Common.CurrentVersion.targets(4364,5): error MSB3371: The file "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-AC-JOB1\test\Integration.Kardex\obj\Release\netcoreapp2.1\win-x64\Integration.Kardex.csproj.CopyComplete" cannot be created. The process cannot access the file 'D:\Atlassian\Bamboo\xml-data\build-dir\TIC-AC-JOB1\test\Integration.Kardex\obj\Release\netcoreapp2.1\win-x64\Integration.Kardex.csproj.CopyComplete' because it is being used by another process. [D:\Atlassian\Bamboo\xml-data\build-dir\TIC-AC-JOB1\test\Integration.Kardex\Integration.Kardex.csproj]

Environment data

dotnet --info output:

.NET Core SDK (reflecting any global.json): Version: 2.1.301 Commit: 59524873d6

Runtime Environment: OS Name: Windows OS Version: 6.3.9600 OS Platform: Windows RID: win81-x64 Base Path: C:\Program Files\dotnet\sdk\2.1.301\

Host (useful for support): Version: 2.1.1 Commit: 6985b9f684

.NET Core SDKs installed: 1.0.4 [C:\Program Files\dotnet\sdk] 1.1.0 [C:\Program Files\dotnet\sdk] 2.0.0 [C:\Program Files\dotnet\sdk] 2.0.2 [C:\Program Files\dotnet\sdk] 2.0.3 [C:\Program Files\dotnet\sdk] 2.1.101 [C:\Program Files\dotnet\sdk] 2.1.201 [C:\Program Files\dotnet\sdk] 2.1.301 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed: Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs: https://aka.ms/dotnet-download

[jez9999]

I second the vote for “make it work somehow”. Telling people they have to manually build every project separately is ridiculous. People expect to just build a solution and for the build engine to work.

[rainersigwald]

From that log:

547257:Project "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\Tic.sln" (1) is building "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\test\Integration.Kardex\Integration.Kardex.csproj" (35) on node 1 (Publish target(s)).
637706:Project "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\test\Integration.Kardex.Wamp.AspNetCore\Integration.Kardex.Wamp.AspNetCore.csproj" (37) is building "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\test\Integration.Kardex\Integration.Kardex.csproj" (35:2) on node 5 (GetTargetFrameworks target(s)).
3075143:Project "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\test\Integration.Kardex.Wamp.AspNetCore\Integration.Kardex.Wamp.AspNetCore.csproj" (37) is building "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\test\Integration.Kardex\Integration.Kardex.csproj" (35:3) on node 2 (default targets).

That indicates that Integration.Kardex.csproj is building with 3 separate sets of global properties. The call to GetTargetFrameworks should be harmless–it should only return information about the project, not do any actual work with files on disk. That leaves the solution reference and the ProjectReference from Integration.Kardex.Wamp.AspNetCore.csproj.

As shown in that image, both of the racing instances of the project actually have the same TF and RID, which is where I expected to see the global property change. Unfortunately, the binary log wasn’t captured with an MSBuild that has https://github.com/Microsoft/msbuild/pull/3252 (fixed in 15.8 that’s not yet released), so it’s pretty hard to figure out what properties are different.

Based on testing on a simplified project (one mvc project referencing one netcoreapp2.1 classlib), I think there’s a race between the project as built by the solution (inheriting the RID and TF passed to the solution by dotnet publish) and the project as built by the reference (with explicitly unset RID and TF, because there’s only one in the file).

But that doesn’t seem to be what’s happening here, because the query-TFs call didn’t collapse into the build-single-TF call. I can’t tell why from the logging; I’m getting lost in the various different properties set in different projects.

There’s definitely a problem here. At the moment I would suggest running publish only on individual projects and not at the solution level. That will help ensure that TargetFramework and RID flow correctly down to referenced projects.