dotnet global tools failing to uninstall or update on 2.1-sdk-alpine

@RyanWhite25 commented on Wed Oct 30 2019

Steps to reproduce the issue

1. Create a basic dockerfile as below

FROM microsoft/dotnet:2.1-sdk-alpine

RUN dotnet tool install -g Amazon.Lambda.Tools

CMD ["/bin/sh"]

1. Build a Docker image and run a container

docker build <dir>
docker container run -it <image id>

1. Attempt to uninstall or update the nuget package

dotnet tool update -g Amazon.Lambda.Tools

dotnet tool uninstall -g Amazon.Lambda.Tools

Expected behavior

The package is updated or uninstalled

Actual behavior

An error message is thrown:

Tool 'amazon.lambda.tools' failed to update due to the following:
Failed to uninstall tool package 'amazon.lambda.tools': Cross-device link

Additional information (e.g. issue happens only occasionally)

I’ve ran into an issue when uninstalling or updating nuget packages on microsoft/dotnet:2.1-sdk-alpine, when baked into a docker image via a dockerfile.

I’ve been able to replicate on different host machines, and only seems to occur when the package has been pre-installed on the image. Running the base container interactively and installing allows me to update and uninstall the package as expected.

I’ve used Amazon.Lambda.Tools as an example, but this seems to also occur on the several other packages I’ve tested.

It also seems to occur when using --tools-path instead of global.

A seemingly similar issue has also been reported on StackOverflow here: https://stackoverflow.com/questions/57792546/docker-uninstall-dotnet-global-tool-installed-through-dockerfile

I’m specifically seeing this issue when trying to build PowerShell Core AWS Lambda functions using New-AWSPowerShellLambdaPackage, which is failing when it can’t update the existing installation of Amazon.Lambda.Tools: https://github.com/aws/aws-lambda-dotnet/blob/master/PowerShell/Module/Private/_DeploymentFunctions.ps1#L314

Thanks for any information or assistance you can provide with this.

Output of docker version

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
 Built:             Mon Jul  1 18:51:44 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a/18.06.1-ce
  Built:            Mon Jul  1 18:53:20 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info

Containers: 20
 Running: 1
 Paused: 0
 Stopped: 19
Images: 28
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.14.138-114.102.amzn2.x86_64
Operating System: Amazon Linux 2
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 983.7MiB
Name: ip-172-31-11-191.eu-west-1.compute.internal
ID: J53Z:IZEG:YQT5:E2KR:SWNZ:JTHZ:QQCE:3ZWZ:VX4M:IDVJ:5W5B:6LG5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

---

@mthalman commented on Thu Oct 31 2019

I’ve been able to repro this issue. I’ve also confirmed that the same error occurs for both 2.2 and 3.0 SDKs.

@RyanWhite25 - Although this is unrelated to the issue you encountered, we recommend that you update your .NET image references to use the MCR Docker registry. In your case, microsoft/dotnet:2.1-sdk-alpine would be replaced with mcr.microsoft.com/dotnet/core/sdk:2.1-alpine. These new tags are all published on Docker Hub.

---

@MichaelSimons commented on Thu Oct 31 2019

This issue is also not specific to the alpine images, you can reproduce it with Debian for example (e.g. mcr.microsoft.com/dotnet/core/sdk:3.0). Given the Cross-device link error, I am speculating this is related to Docker layering and the overlay mount that is created that unions all of the image layers together. For example if you install and uninstall the tool within a single layer everything works fine. Does the dotnet tool infrastructure rely on hard links? Moving to the SDK team to investigate.

[MichaelSimons]

Running this scenario with diagnostic output yields the following:

Microsoft.DotNet.ToolPackage.ToolPackageException: Invalid cross-device link
 ---> System.IO.IOException: Invalid cross-device link
   at System.IO.FileSystem.MoveDirectory(String sourceFullPath, String destFullPath)
   at System.IO.Directory.Move(String sourceDirName, String destDirName)
   at ToolPackageUninstaller.<>c__DisplayClass2_1.<Uninstall>b__3()
   at Microsoft.DotNet.Cli.Utils.FileAccessRetrier.RetryOnMoveAccessFailure(Action action)
   at ToolPackageUninstaller.<>c__DisplayClass2_0.<Uninstall>b__0()
   --- End of inner exception stack trace ---
   at ToolPackageUninstaller.<>c__DisplayClass2_0.<Uninstall>b__0()
   at Microsoft.DotNet.Cli.TransactionalAction.<>c__DisplayClass2_0.<Run>b__0()
   at Microsoft.DotNet.Cli.TransactionalAction.Run[T](Func`1 action, Action commit, Action rollback)
   at Microsoft.DotNet.Cli.TransactionalAction.Run(Action action, Action commit, Action rollback)
   at ToolPackageUninstaller.Uninstall(DirectoryPath packageDirectory)
   at Microsoft.DotNet.Tools.Tool.Uninstall.ToolUninstallGlobalOrToolPathCommand.Execute()
Failed to uninstall tool package 'amazon.lambda.tools': Invalid cross-device link

It looks like this is caused by the logic to move the global tool package directory to a staging directory - https://github.com/dotnet/cli/blob/release/3.0.1xx/src/dotnet/ToolPackage/ToolPackageUninstaller.cs#L37

Directory.Move does not support mounts on linux - Per the documentation IOException An attempt was made to move a directory to a different volume.

There is at least one open issue request Directory.Move mount support - https://github.com/dotnet/corefx/issues/41734

[MichaelSimons]

@wli3 - Docker creates an overlay mount which merges all of the Docker image layers.

Am I wrong in thinking that the best solution here is for Directory.Move to support mounts (e.g. https://github.com/dotnet/corefx/issues/41734)?

From a priority perspective, the scenario involving uninstalling a dotnet tool that was previously installed in a Docker layer seems like a low priority. The scenario involving upgrading a dotnet tool that was installed in a previous Docker layer is higher in priority but I am not sure how common that would be either. I’m speaking from a scenario perspective here, ignoring the fact that upgrade tool implementation relies on uninstalling the previous version.