GenerateDepsFile: The process cannot access the file '...\MyProject.deps.json' because it is being used by another process.
See original GitHub issueUPDATE: While writing this, rather long, post – I’ve managed to produce a small reproduction example, see bottom
We’ve recently upgraded our builds to run on faster hardware with more cpu cores, and are now ~100% of the time seeing the below error in a number of repositories. The error here is from Windows, but our CI servers run on linux and have the same error (see logs below).
We run the following commandlines:
“C:\Program Files\dotnet\dotnet.exe” restore C:\Project\MyProject.sln “C:\Program Files\dotnet\dotnet.exe” build C:\Project\MyProject.sln --configuration Debug --framework netstandard2.0 --no-restore
C:\Program Files\dotnet\sdk\2.2.101\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.targets(129,5): error MSB4018
The "GenerateDepsFile" task failed unexpectedly. [C:\Project\MyProject.csproj]
System.IO.IOException: The process cannot access the file 'C:\Project\bin\Debug\netstandard2.0\MyProject.deps.json' because it is being used by another process. [C:\Project\MyProject.csproj]
at System.IO.FileStream.ValidateFileHandle(SafeFileHandle fileHandle) [C:\Project\MyProject.csproj]
at System.IO.FileStream.CreateFileOpenHandle(FileMode mode, FileShare share, FileOptions options) [C:\Project\MyProject.csproj]
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) [C:\Project\MyProject.csproj]
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize) [C:\Project\MyProject.csproj]
at System.IO.File.Create(String path) [C:\Project\MyProject.csproj]
at Microsoft.NET.Build.Tasks.GenerateDepsFile.ExecuteCore() [C:\Project\MyProject.csproj]
at Microsoft.NET.Build.Tasks.TaskBase.Execute() [C:\Project\MyProject.csproj]
at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [C:\Project\MyProject.csproj]
at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [C:\Project\MyProject.csproj]
Observations:
- This fails on my local Windows desktop roughly 33% of the time. Rerunning the build (no cleanup) always succeeds (seems to be a race condition, so HW speeds play a role). On our Linux CI servers, it fails 100% of the time currently
- The deps file in question is always the same file for a given repository / builds. The file is for a shared library between multiple projects.
- A plain
dotnet build MyProject.sln -f netstandard2.0
(no build script) also exhibits this issue. - A
dotnet build MyProject.sln
seemingly always succeeds (locally and on CI server), at least for three consecutive attempts right now. - All the projects in the
MyProject.sln
havenetstandard2.0
- but a number of them have more targets (ie. netcoreapp) - ~We have more projects, f.ex. for tests, but they’re in
MyProject-Tests.sln
. These projects only targetnetcoreapp2.x
- we’ve split the solutions both for speed when developing and to be able to build artifacts without compiling the tests (since we can’t build certain projects from the solution only … that’s another issue entirely)~ (not relevant) - Limiting the build to one process (msbuild:
/m:1
) seems to “solve” the problem. - At work (inaccessible currently), I performed some Procmon captures to see the difference between successful and failing runs
- It turns out, that two
dotnet.exe
processes attempt to create the.deps.json
files - When succeeding, there is a clear seperation between two CreateFile() calls, where the first creates the file, and the second call (from the second PID) finds the file already there and does nothing (not even reads it)
- When failing, the two processes make calls overlapping each other, leading to both of them discovering the file as missing, and both trying to create it (one obviously failing, producing the stacktrace above) – classical concurrency issue.
- It turns out, that two
So.
- It should be that when two projects reference a third, shared, project … that this shared one is built exactly once … right?.. How come two attempts are made at writing the same
.deps.json
files? - How come this seemingly works fine, when building for all target frameworks?
Versions:
- Local, Windows 10 x64,
dotnet 2.2.101
- Linux CI, docker,
dotnet 2.2.103
(we build our own docker images using the dockerfile from the basemicrosoft/dotnet:2-sdk
images)
Reproduction
Project: ci-stresstest-master.zip
In it, I have a solution with 5 projects, all empty (no actual code), but with a targeting setup that mimicks a very small subset of our projects. I’ve then created four seperate CI builds that did the following:
- log (succeeds)
dotnet restore ci-stresstest.sln
+dotnet build ci-stresstest.sln --no-restore
- log (succeeds)
dotnet build ci-stresstest.sln
- log (fails)
dotnet restore ci-stresstest.sln
+dotnet build ci-stresstest.sln -f netstandard2.0 --no-restore
- log (fails)
dotnet build ci-stresstest.sln -f netstandard2.0
Notes on repro.
- We run a vanilla docker setup for CI, except for one thing:
- We mount a shared directory for nuget packages, to be able to share downloaded packages between builds
See also #2076
Issue Analytics
- State:
- Created 5 years ago
- Reactions:28
- Comments:22 (6 by maintainers)
After switching/adding on TargetFrameworks (
net6.0;net7.0
),msbuild
immediately starting giving file access concurency warnings with retries, and frequently started failing entirely. (similar symptoms stated above)In this instance we only had a single solution file, and single project, targeting >1 SDK for build.
The only workaround that completely resolved the issue was to drop down CPU count to 1 to disable any parallel execution.
^ after doing this, ALL warnings were immediately eliminated and build hasn’t failed since
I am getting this error if I simply call
dotnet build Main.sln
without specifying the target framework. So I can’t believe in this “core problem” explanation.