Preview 7 UX for Arm64/x64 co-existence on macOS

Let’s take a look at the experience that is offered for Arm64/x64 coexistence on macOS.

I will use this app for the test. https://github.com/dotnet/core/tree/main/samples/dotnet-runtimeinfo

Installing .NET (correctly)

We haven’t updated the installers yet, so I had to take some special steps to setup my machine as if they did. Here’s what I did.

• Delete .NET from my machine
• Install various x64 .NET versions (whichever ones I want).
• Move all the files in dotnet to dotnet\x64
• Install .NET 6 Arm64.

That gets me a working install.

Here are the actual commands I used (skipping over the .pkg installs).

rich@MacBook-Air-M1-2020 share % pwd
/usr/local/share
rich@MacBook-Air-M1-2020 share % sudo -r dotnet
# install x64 versions
rich@MacBook-Air-M1-2020 share % sudo mv dotnet x64
rich@MacBook-Air-M1-2020 share % sudo mkdir dotnet
rich@MacBook-Air-M1-2020 share % sudo mv x64 dotnet
# install .NET 6 Arm64

Running Arm64 apps

rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % pwd 
/Users/rich/git/core/samples/dotnet-runtimeinfo
rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % cat dotnet-runtimeinfo.csproj | grep Target
    <TargetFramework>net6.0</TargetFramework>
rich@MacBook-Air-M1-2020 dotnet-runtimeinfo %
rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % dotnet run
**.NET information
Version: 6.0.0
FrameworkDescription: .NET 6.0.0-preview.7.21377.19
Libraries version: 6.0.0-preview.7.21377.19
Libraries hash: 91ba01788d4d83475fec3aea7c830376e08585da

**Environment information
OSDescription: Darwin 21.0.0 Darwin Kernel Version 21.0.0: Thu Jul 22 19:52:32 PDT 2021; root:xnu-8019.0.72.141.3~2/RELEASE_ARM64_T8101
OSVersion: Unix 12.0.0
OSArchitecture: Arm64
ProcessorCount: 8

That’s good. That’s expected.

Running x64 apps – take one

rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % pwd
/Users/rich/git/core/samples/dotnet-runtimeinfo
rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % cat dotnet-runtimeinfo.csproj | grep Target
    <TargetFramework>net5.0</TargetFramework>
rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % 
rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % dotnet run   
Failed to load /usr/local/share/dotnet/host/fxr/6.0.0-preview.7.21377.19/libhostfxr.dylib, error: dlopen(/usr/local/share/dotnet/host/fxr/6.0.0-preview.7.21377.19/libhostfxr.dylib, 0x0001): tried: '/usr/local/share/dotnet/host/fxr/6.0.0-preview.7.21377.19/libhostfxr.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/libhostfxr.dylib' (no such file), '/usr/lib/libhostfxr.dylib' (no such file)
The library libhostfxr.dylib was found, but loading it from /usr/local/share/dotnet/host/fxr/6.0.0-preview.7.21377.19/libhostfxr.dylib failed
  - Installing .NET prerequisites might help resolve this problem.
     https://go.microsoft.com/fwlink/?linkid=2063366

Oooh. You’ll see that the .NET 5.0 app is being built as x64. That’s good. That’s an improvement over earlier .NET 6 builds.

Nothing in the system yet knows AFAIK that it should be looking for .NET 6 in the x64 directory. Let’s try some more tricks (the equivalent of what will be automatic).

rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % export DOTNET_ROOT=/usr/local/share/dotnet/x64
rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % ./bin/Debug/net5.0/dotnet-runtimeinfo
**.NET information
Version: 5.0.9
FrameworkDescription: .NET 5.0.9
Libraries version: 5.0.9
Libraries hash: 208e377a5329ad6eb1db5e5fb9d4590fa50beadd

**Environment information
OSDescription: Darwin 21.0.0 Darwin Kernel Version 21.0.0: Thu Jul 22 19:52:32 PDT 2021; root:xnu-8019.0.72.141.3~2/RELEASE_ARM64_T8101
OSVersion: Unix 11.0.0
OSArchitecture: X64
ProcessorCount: 8

That’s good.

Running x64 apps – take two

Let’s apply the new guidance: https://github.com/dotnet/sdk/issues/19743#issuecomment-899094680.

rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % dotnet --info
.NET SDK (reflecting any global.json):
 Version:   6.0.100-preview.7.21379.14
 Commit:    22d70b47bc

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  12.0
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/6.0.100-preview.7.21379.14/

Host (useful for support):
  Version: 6.0.0-preview.7.21377.19
  Commit:  91ba01788d

.NET SDKs installed:
  6.0.100-preview.7.21379.14 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.0-preview.7.21378.6 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.0-preview.7.21377.19 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % cat /etc/dotnet/install_location 
/usr/local/share/dotnet/x64
arm64=/usr/local/share/dotnet
x64=/usr/local/share/dotnet/x64
rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % rm -r obj bin
rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % cat dotnet-runtimeinfo.csproj 
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <Description>Displays .NET version and environment information.</Description>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
    <Nullable>enable</Nullable>
    <RollForward>LatestMajor</RollForward>
    <VersionPrefix>1.0.4</VersionPrefix>
    <PackageProjectUrl>https://github.com/dotnet/core/blob/main/samples/dotnet-runtimeinfo/README.md</PackageProjectUrl>
    <PackageLicenseExpression>MIT</PackageLicenseExpression>
    <Copyright>.NET Foundation</Copyright>
    <Authors>.NET Team</Authors>
    <PackAsTool>true</PackAsTool>
  </PropertyGroup>

</Project>
rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % dotnet run
Failed to load /usr/local/share/dotnet/host/fxr/6.0.0-preview.7.21377.19/libhostfxr.dylib, error: dlopen(/usr/local/share/dotnet/host/fxr/6.0.0-preview.7.21377.19/libhostfxr.dylib, 0x0001): tried: '/usr/local/share/dotnet/host/fxr/6.0.0-preview.7.21377.19/libhostfxr.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/libhostfxr.dylib' (no such file), '/usr/lib/libhostfxr.dylib' (no such file)
The library libhostfxr.dylib was found, but loading it from /usr/local/share/dotnet/host/fxr/6.0.0-preview.7.21377.19/libhostfxr.dylib failed
  - Installing .NET prerequisites might help resolve this problem.
     https://go.microsoft.com/fwlink/?linkid=2063366
rich@MacBook-Air-M1-2020 dotnet-runtimeinfo % ./bin/Debug/net5.0/dotnet-runtimeinfo
**.NET information
Version: 5.0.9
FrameworkDescription: .NET 5.0.9
Libraries version: 5.0.9
Libraries hash: 208e377a5329ad6eb1db5e5fb9d4590fa50beadd

**Environment information
OSDescription: Darwin 21.0.0 Darwin Kernel Version 21.0.0: Thu Jul 22 19:52:32 PDT 2021; root:xnu-8019.0.72.141.3~2/RELEASE_ARM64_T8101
OSVersion: Unix 11.0.0
OSArchitecture: X64
ProcessorCount: 8

Looks like we have the right apphost and it is doing the right thing but that the CLI is not with dotnet run. I tried adding the UseAppHost property but that didn’t change the behavior.

Using the Arm64 SDK to target x64

That’s actually what I demonstrated above.

I tried using the new -a syntax, separately. It doesn’t quite work correctly yet. See: https://github.com/dotnet/sdk/issues/19706

Installing tools

Let’s see how .NET tools works.

rich@MacBook-Air-M1-2020 ~ % dotnet tool list -g
Package Id      Version      Commands
rich@MacBook-Air-M1-2020 ~ % dotnet tool install -g dotnet-runtimeinfo
rich@MacBook-Air-M1-2020 ~ % export PATH="$PATH:/Users/rich/.dotnet/tools"
rich@MacBook-Air-M1-2020 ~ % dotnet-runtimeinfo                           
zsh: killed     dotnet-runtimeinfo
rich@MacBook-Air-M1-2020 ~ % which dotnet-runtimeinfo
/Users/rich/.dotnet/tools/dotnet-runtimeinfo
rich@MacBook-Air-M1-2020 ~ % codesign -s - /Users/rich/.dotnet/tools/dotnet-runtimeinfo
rich@MacBook-Air-M1-2020 ~ % dotnet-runtimeinfo       
**.NET information
Version: 6.0.0
FrameworkDescription: .NET 6.0.0-preview.7.21377.19
Libraries version: 6.0.0-preview.7.21377.19
Libraries hash: 91ba01788d4d83475fec3aea7c830376e08585da

**Environment information
OSDescription: Darwin 21.0.0 Darwin Kernel Version 21.0.0: Thu Jul 22 19:52:32 PDT 2021; root:xnu-8019.0.72.141.3~2/RELEASE_ARM64_T8101
OSVersion: Unix 12.0.0
OSArchitecture: Arm64
ProcessorCount: 8
rich@MacBook-Air-M1-2020 ~ % cat /Users/rich/.dotnet/tools/.store/dotnet-runtimeinfo/1.0.4/dotnet-runtimeinfo/1.0.4/tools/netcoreapp3.1/any/dotnet-runtimeinfo.runtimeconfig.json 
{
  "runtimeOptions": {
    "tfm": "netcoreapp3.1",
    "rollForward": "LatestMajor",
    "framework": {
      "name": "Microsoft.NETCore.App",
      "version": "3.1.0"
    }
  }
}
                                              

I was expecting this app to run on x64 since it targets .NET 3.1. I didn’t validate this with an app that doesn’t specify LatestMajor rollforward, but I assume that property is not being consulted/honored.

At present, I’m consider this incorrect behavior. That’s a dupe of https://github.com/dotnet/sdk/issues/17241.

[vitek-karas]

As I suspected (from the log):

Using environment variable DOTNET_ROOT=[/usr/local/share/dotnet] as runtime location.

As @matoatr pointed out offline, we will need to modify this: https://github.com/dotnet/sdk/blob/main/src/Cli/dotnet/commands/dotnet-run/RunCommand.cs#L257

I discussed this with @mateoatr and we think the correct behavior will be:

• If the app targets the same architecture as SDK is running on - SDK should continue what it’s doing currently (set DOTNET_ROOT).
  • Ideally for TFM 6+ SDK should set DOTNET_ROOT_arch only in this case (would make things much cleaner going forward) - but it would work without this.
• If the app targets different architecture from the SDK, SDK should NOT set anything (since it doesn’t know, so it should not try).

[vitek-karas]

@mateoatr Can you please take a look what’s going on with the dotnet run - trying to run .NET 5 app (x64) from .NET 6 SDK (arm64). My suspicion is that it sets DOTNET_ROOT to point to the SDK’s version of the runtime, which is wrong in this case.

@richlander - can you please try this (I don’t have amr64 mac to try this on): In the state right before dotnet run

export COREHOST_TRACE=1
export COREHOST_TRACEFILE=host.txt
dotnet run

And then share the host.txt file (note that it may contain environment, path and other information about machine/user you’re running this on, so make sure it’s OK to share publicly).