Microsoft.WindowsAzure.targets in combination with PackageReference resolves ref target

When Cloud Service Solution with Microsoft.WindowsAzure.targets is used in combination with PackageReference (old csproj format with package reference interop mode) in combination with a reference assembly like System.ValueTuple the reference assembly is copied into the output path instead of the implementation assembly for the target framework.

Prerequisites

Azure SDK 2.9 VS2017 15.3.3 or higher

Steps to reproduce

• Create a new cloud services solution
• Add a new worker role, target framework < 4.7 (preferably 4.6.2 or smaller)
• Reference System.ValueTuple from nuget via packages.config
• Add

        private static (int foo, bool bar) Read()
        {
            return (42, true);
        }

• Use method in Run like Trace.TraceInformation(Read().ToString());
• Debug locally with F5
• Works see how csx\Debug\roles\WorkerRole1\approot contains 78 kb size ValueTuple implemenation assembly
• Add another worker role, target framework < 4.7 (preferably 4.6.2 or smaller)
• Remove packages.config
• Reference System.ValueTuple via package references

  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.KeyVault.Core" Version="2.0.4" />
    <PackageReference Include="Microsoft.Data.Edm" Version="5.*" />
    <PackageReference Include="Microsoft.Data.OData" Version="5.*" />
    <PackageReference Include="Microsoft.Data.Services.Client" Version="5.*" />
    <PackageReference Include="Microsoft.WindowsAzure.ConfigurationManager" Version="3.*" />
    <PackageReference Include="Newtonsoft.Json" Version="8.*" />
    <PackageReference Include="System.Spatial" Version="5.*" />
    <PackageReference Include="System.ValueTuple" Version="4.*" />
    <Reference Include="Microsoft.WindowsAzure.Diagnostics, Version=2.8.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35">
      <Private>True</Private>
    </Reference>
    <Reference Include="Microsoft.WindowsAzure.ServiceRuntime, Version=2.7.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35">
      <Private>False</Private>
    </Reference>
    <PackageReference Include="WindowsAzure.Storage" Version="8.4.0" />
    <Reference Include="Microsoft.CSharp" />
    <Reference Include="System.Data" />
    <Reference Include="System.Xml" />
  </ItemGroup>

• Add

        private static (int foo, bool bar) Read()
        {
            return (42, true);
        }

• Use method in Run like Trace.TraceInformation(Read().ToString());
• Debug locally with F5
• Observe BadImageFormatException

Could not load file or assembly 'System.ValueTuple, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)

   at WorkerRole2.WorkerRole.Run()
   at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.StartRoleInternal()
   at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.<StartRole>b__2()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

• Works see how csx\Debug\roles\WorkerRole2\approot contains 40 kb size ValueTuple ref assembly without implemenations

Repro Solution https://github.com/danielmarbach/PackageRef.CloudServicesRepro

Expected behavior

Target copies implementation assembly instead of ref assembly to csx output path.

[yirkha]

Hi all,

Analysis

In case of a NuGet package that provides both kinds of assemblies, the build-only reference assemblies from ref/ end up in a MSBuild property @(ReferencePath) and the full runtime assemblies from lib/ end up in @(ReferenceCopyLocalPaths).

A target CollectWorkerRoleFiles in Microsoft.WindowsAzure.targets later queries all binaries to copy and package from the individual role projects by asking the dependent project to “build” a target BuiltProjectOutputGroup from Microsoft.Common.CurrentVersion.targets:

<Target
    Name="BuiltProjectOutputGroupDependencies"
    DependsOnTargets="$(BuiltProjectOutputGroupDependenciesDependsOn)"
    Returns="@(BuiltProjectOutputGroupDependency)">
  <ItemGroup>
    <BuiltProjectOutputGroupDependency
                Include="@(ReferencePath->'%(FullPath)');
                         @(ReferenceDependencyPaths->'%(FullPath)');
                         @(NativeReferenceFile->'%(FullPath)');
                         @(_DeploymentLooseManifestFile->'%(FullPath)');
                         @(ResolvedIsolatedComModules->'%(FullPath)')"/>
  </ItemGroup>
</Target>

As you can see, this includes @(ReferencePath) in the result, not @(ReferenceCopyLocalPaths). Therefore it will provide the reference assemblies only and result in a broken final package with incorrect dependencies.

Solution

Apparently BuiltProjectOutputGroupDependencies should not be used in these cases and ReferenceCopyLocalPathsOutputGroup should be used instead. See https://github.com/microsoft/msbuild/issues/3069

So the fix should happen in Microsoft.WindowsAzure.targets, which is unfortunately not a part of this open source Azure SDK for .NET project here on GitHub. It comes from what’s called “Azure Cloud Services build tools” component of Visual Studio and developed internally at Microsoft (DevDiv/AT-OneSdk).

I have no idea how to raise a bug against that, developercommunity.visualstudio.com maybe?

Workaround

Adding this at the end of the inner .csproj files (for the individual worker roles) fixed the issue for me. A side effect is that it resulted in including some auxiliary .xml files too, maybe it could be made less dumb.

  <!-- When the final Azure Cloud Service project is retrieving all binaries to include in the deployment package,
       it mistakenly queries the BuiltProjectOutputGroupDependencies target in this project, which doesn't contain
       the full runtime assemblies from NuGet packages that have both the lib/ and ref/ directories. So override
       the target here to include ReferenceCopyLocalPaths as a workaround.
       See also https://github.com/Azure/azure-sdk-for-net/issues/3699 -->
  <Target
      Name="BuiltProjectOutputGroupDependencies"
      DependsOnTargets="$(BuiltProjectOutputGroupDependenciesDependsOn)"
      Returns="@(BuiltProjectOutputGroupDependency)">
    <ItemGroup>
      <BuiltProjectOutputGroupDependency Include="@(ReferenceCopyLocalPaths->'%(FullPath)');
                                                  @(ReferenceDependencyPaths->'%(FullPath)');
                                                  @(NativeReferenceFile->'%(FullPath)');
                                                  @(_DeploymentLooseManifestFile->'%(FullPath)');
                                                  @(ResolvedIsolatedComModules->'%(FullPath)')"/>
    </ItemGroup>
  </Target

[mbihler]

If someone has missing COM wrapper assemblies when using above workaround: Add this to the include list: @(ReferenceComWrappersToCopyLocal->'%(FullPath)')