libgit2sharp cannot find native dependencies in linux container, but only when hosted by Azure DevOps

On this Azure DevOps build, the LinuxContainer job failed with

2018-09-13T20:16:21.4909762Z          /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018: The "Nerdbank.GitVersioning.Tasks.GetBuildVersion" task failed unexpectedly. [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.4930956Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018: System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. ---> System.DllNotFoundException: Unable to load shared library '/home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/lib/linux/x86_64/libgit2-6311e88.so' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libcurl-gnutls.so.4: cannot open shared object file: No such file or directory [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.4959306Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at System.Runtime.Loader.AssemblyLoadContext.InternalLoadUnmanagedDllFromPath(String unmanagedDllPath) [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.4975704Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at System.Runtime.Loader.AssemblyLoadContext.LoadUnmanagedDllFromPath(String unmanagedDllPath) [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.4991894Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at MSBuildExtensionTask.ContextAwareTask.CustomAssemblyLoader.LoadUnmanagedDll(String unmanagedDllName) [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.5008294Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at System.Runtime.Loader.AssemblyLoadContext.ResolveUnmanagedDll(String unmanagedDllName, IntPtr gchManagedAssemblyLoadContext) [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.5024432Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.Core.NativeMethods.git_libgit2_init() [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.5040121Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.Core.NativeMethods.LoadNativeLibrary() [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.5056062Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.Core.NativeMethods..cctor() [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.5073320Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    --- End of inner exception stack trace --- [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.5090767Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.Core.NativeMethods.git_libgit2_opts(Int32 option, UInt32 level, String path) [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.5107307Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at LibGit2Sharp.GlobalSettings.SetConfigSearchPaths(ConfigurationLevel level, String[] paths) [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.5123634Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at Nerdbank.GitVersioning.GitExtensions.OpenGitRepo(String pathUnderGitRepo) [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.5139915Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at Nerdbank.GitVersioning.VersionOracle.Create(String projectDirectory, String gitRepoDirectory, ICloudBuild cloudBuild, Nullable`1 overrideBuildNumberOffset, String projectPathRelativeToGitRepoRoot) [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]
2018-09-13T20:16:21.5156505Z        /home/vsts_VSTSContainer/.nuget/packages/nerdbank.gitversioning/2.2.13/build/Nerdbank.GitVersioning.targets(63,5): error MSB4018:    at Nerdbank.GitVersioning.Tasks.GetBuildVersion.ExecuteInner() [/__w/1/s/src/StreamJsonRpc/StreamJsonRpc.csproj]

What’s particularly odd to me is that this step is running within a linux container, based on an image I have tested locally and confirmed that it can in fact build the code without error using these commands locally:

# paste this first
docker run --rm -it andrewarnott/linux-buildagent /bin/bash

# then paste this after the docker container terminal is ready
apt update
apt-get install -y git

# finally, paste this
git clone https://github.com/microsoft/vs-streamjsonrpc
cd vs-streamjsonrpc
git checkout 9c81434121d21b3b4d457fa2dfde5154bb519f92
dotnet build src/StreamJsonRpc.Tests/StreamJsonRpc.Tests.csproj -f netcoreapp1.0

I thought the whole value-prop of linux containers was that this kind of inconsistency would not happen.

This may not be a bug in libgit2sharp, but @bording has been very helpful in diagnosing these bugs before, and I’m hoping he can again. 😄 🙏

[AArnott]

Yay!

The missing package was libcurl3-gnutls.

I first repro’d the failure locally by mapping in an existing git clone.

Then I looked at the packages that installing git would have installed and found just the one curl-related one mentioned above. So I installed that directly instead of installing git. Build then succeeded.

[AArnott]

The first difference I see is that the passing version of the log is using Nerdbank.GitVersioning 2.2.10 and the failing one is using 2.2.13.

I’m sorry. I must have sent the wrong “good” file. I just looked at what I sent you and it mentions /home/andrew which is not in the container. The “good” container very clearly is using 2.2.13 as that is the only version that has been restored. Here is the real “good container” LD_DEBUG log. So sorry: ld_good.zip

But hey, maybe I can save you some time redoing work. You mentioned libcurl. I am installing libcurl3 here (I think that was included in the image we’re testing). But what is different between the hosted and local container is that I installed (wait for it…) git on the local container so I could clone into it. The hosted build doesn’t need this because the clone happens external to the container and that folder gets mapped in.

So ya, I’ll work on a repro without installing git locally and install packages till it works. I should have thought of that before reaching out. Sorry.