Usage of hard or symbolic linking leads to NuGet cache corruption

Issue Description

MSBuild can use hard or symbolic links to avoid excessive file copies on disk. It is a good method to speed-up builds but at the same time, it is very easy to silently corrupt the NuGet cache. I have seen that this issue has been mentioned several times already, but it was never explained how and when exactly the NuGet cache gets corrupted:

Related issues:

• https://github.com/dotnet/msbuild/issues/1407
• https://github.com/dotnet/msbuild/issues/3788
• https://github.com/dotnet/msbuild/issues/7014
• https://github.com/NuGet/Home/issues/12047

Steps to Reproduce

1. Build an executable project with hard or symbolic links enabled
2. Update the version of referenced NuGet package
3. Build the application again, but this time, without using hard/symbolic links. MSBuild will try to update dependencies in the application’s output folder. Sadly, instead of replacing existing links, it replaces actual files in NuGet cache (thus corrupting it).

• hard-links

dotnet nuget locals --clear all
dotnet new console
dotnet add package newtonsoft.json -v 13.0.1
dotnet build /p:CreateHardLinksForCopyLocalIfPossible=true
dotnet add package newtonsoft.json -v 13.0.2
dotnet build

• symbolic-links

dotnet nuget locals --clear all
dotnet new console
dotnet add package newtonsoft.json -v 13.0.1
dotnet build /p:CreateSymbolicLinksForCopyLocalIfPossible=true
dotnet add package newtonsoft.json -v 13.0.2
dotnet build

In both cases file newtonsoft.json\13.0.1\lib\netstandard2.0\Newtonsoft.Json.dll is silently replaced with newtonsoft.json\13.0.2\lib\net6.0\Newtonsoft.Json.dll:

Expected Behavior

Files in the NuGet cache remain untouched.

Actual Behavior

Files in a NuGet package are silently replaced with files from another version.

Analysis

Both Windows and Linux systems are affected, also it doesn’t matter whether hard or symbolic links are used. The problem is that File.Copy operation, instead of replacing the link, replaces the file that the link is linking to. To safely replace a link with a different file or link, File.Delete needs to be called first. Unfortunately, MSBuild calls File.Delete only when the usage of hard or symbolic links is requested. When the build doesn’t use hard or symbolic links, then the File.Delete is not called.

Versions & Configurations

[JeremyKuhne]

GetFullPath is still significantly expensive, right? Although perhaps better than before, it will still hit the disk in some cases, right?

@danmoseley On Windows it typically is just a string parsing/manipulation routine. It will get the current directory and possibly environment variables (for drive relative paths), but those are not disk based. There is some sort of check regarding legacy device names PRN, CON, etc. but I don’t recall precisely what it does off the top of my head (check for device availability for serial ports maybe?).

On .NET Framework (and early .NET Core versions) Path.GetFullPath was significantly more expensive as we would try to parse the path and check for validity up front (before passing it to the OS). In 4.6.2 I ported back some of the changes which improves things dramatically.

[Forgind]

@Forgind The alternative I had was to replace the string.Equals on the specified paths in DoCopyIfNecessary with a PathsAreIdentical. Is Path.GetFullPath that expensive?

I see that is what you did in #8685.

Right; I couldn’t think of another way to validate that the paths really are the same, but I moved the check a little earlier and tried to unify some of the computation to minimize the impact in most cases. The only case where I think perf regresses is if you try to create a hard/symlink, and that’s minor, per JeremyKuhne’s comment, unless someone is using an older runtime. I undrafted the PR; from internal discussion, it sounds like we’re leaning towards taking it, so that should hopefully resolve the writing-through-symlink problem without reintroducing the copy-onto-self-means-delete problem.