Add Warning? dotnet publish --output of multiple projects corrupts dependencies (DLL hell)

Steps to reproduce

Build two projects with the same output:

dotnet publish --output <absolutepath> A.csproj
dotnet publish --output <absolutepath> B.csproj

Expected behavior

Errors when the A and B projects attempt to copy conflicting versions of some dependency (Foo.dll).

OR

Tells us that we’re being very naughty and not to ever publish two projects to the same directory.

Actual behavior

B’s version of Foo.dll clobbers the second. Executables installed by A can run with undefined behavior because they pick up the wrong version of this library (reviving DLL hell).

Environment data

dotnet --info output:

We’re releasing a project that is testing on Mac, Windows, Linux and inside Docker containers. This is consistent across all the versions of the .NET core SDK we’re using (oldest 2.1.202 under Docker, newest is 2.1.500 on Azure DevOps Pipelines hosts). Here’s one example from an ADO run:

 .NET Core SDK (reflecting any global.json):
Version:   2.1.500
Commit:    b68b931422
Runtime Environment:
OS Name:     ubuntu
OS Version:  16.04
OS Platform: Linux
RID:         ubuntu.16.04-x64
Base Path:   /usr/share/dotnet/sdk/2.1.500/

Host (useful for support):
Version: 2.1.6
Commit:  3f4f8eebd8

[dasMulli]

@livarcocc is there a tracking issue for publishing on .sln files? I’ve seen some more issues over the last month involving publishing on solution files.

A lot of them just have a DotNetCoreCLI@2 azure devops yml task with command: publish that they copied from somewhere and then things start to go wrong when the project grows.

Not sure if there are any plans for helping in this scenario - personally I’d be in favor of a breaking change for 3.0 since in many cases, the users have been in dangerous waters before.

[rrnewton]

This is related to the following issues:

• publish multiple projects at once: https://github.com/dotnet/cli/issues/4825
• deduplicating the output when publishing a multi-project solution: https://github.com/dotnet/cli/issues/10119
• clean output folder: https://github.com/dotnet/cli/issues/2855
• (tangentially) to other examples of stomping output (multi target): https://github.com/dotnet/cli/issues/2216

Background

I’m currently on a team at Microsoft Research that is releasing an open source project. In spite of the fact that most everyone involved in the project is a C#/.NET veteran, they were used to Visual Studio, and when trying the dotnet CLI it “seemed fine” to publish four projects to the same output directory.

Indeed, it gives a false sense of confidence, because if the projects have coherent dependencies, then wow, the dependencies appear shared/deduplicated like dotnet CLI is doing something “smart” (as requested by dotnet/sdk#9816). But actually, the files are just stomping each other and we saw bugs as soon as any version of a dependencie (Remote.Linq.dll in our case) gets clobbered by a conflicting version.

Observations

It takes a particular set of policies for this to become a silent failure / corruption. It could be avoided in several ways:

• IF the output directory were simply cleaned every time (#2855), we would have immediately noticed our mistake, because all publishes but the last would have disappeared.

• IF there were a “publish --fragile” mode, it could check before overwriting a file and error if the files were not identical. (That way, we could clean the destination first, then built multiple projects and it would succeed if and only if they have perfectly coherent dependencies.)

• IF there were a notion of multi-project publish (#4825) then it would have either dealt with this (separate subdirectories for dependencies, perhaps deduplicated) or would have errored in a manner similar to the “–fragile” solution.

Finally, dotnet publish could simply have had heurustics to try to “guess” if it looks like we’re publishing multiple projects on top of each other, and yell at us. (This is surely the most backwards compatible approach.)

Temporary solution

Once we realized this mistake, we needed a deduplication approach as requested by dotnet/sdk#9816. For that I have created a hacky Bash script that works for Mac OS and Linux, found in this GIST. It symlinks the duplicated dependency files between different subfolders, and we also symlink the executables themselves to the top of the publish destination so there’s only one folder to add to $PATH.

But what about Windows? The symlink story is a mess there and I don’t see a good deduplication option. (I’m not even sure how to link/forward bin/a.exe to bin/a_subdir/a.exe in a way that picks up the .DLL dependencies inside the subdir.)