RFC: dotnet [Arm64 | x64] coexistence

Microsoft and Apple are both producing new OSes for Arm64 that support x64 emulation. For .NET, we need to determine how to support .NET Arm64 and x64 builds on those OSes in a way that offers a reasonable user experience and that has reasonable cost.

Problem to solve

As stated, we need to offer x64 and Arm64 builds of .NET within one 64-bit namespace (without much help from the OS on how to manage it). That leads to three issues:

• How do we separate the products in some directory structure (peer directories or unified)?
• Do customers manage using the arm64 vs arm64 product via which one is in the path (or absolute paths to dotnet)?
• Do the directory names represent the product in the way we want?

One could reasonable ask why we need to support coexistence once native Arm64 build are available.

• We are only building .NET 6 for Apple Arm64, and people will have legitimate need to use earlier .NET versions that are x64-only.
• The situation on Windows is more favorable since both .NET 5 and 6 are built for Windows Arm64, but the general problem is the same.
• The point of emulation is to enable a smooth transition to Arm64. It may take some time for people to transition to Arm64. For example, someone may be able to move to .NET 6 immediately but not to Arm64 because of a native dependency.
• If we decided not to solve this problem, we’d want to consider blocking the x64 installers on macOS and Windows Arm64 machines, or at least showing a warning that there are unsolved problems that the user will likely run into.

Context

The Windows and macOS plans and implementations are similar but have key differences, per our understanding.

- Same: No WoW64 subsystem or Program Files (x64) style experience. For example, there is no x64 command prompt.
- Same: An executable can be universal.
- Different: macOS universal executables are transparently restartable, while Windows ones are not.
- Different: Windows x64 emulation is here to stay, while macOS x64 emulation will probably be removed in 3-5 years.

The following are various high-level solutions that we could employ, with pros and cons.

Status quo

On Windows x64, .NET is installed to two locations (depending on architecture):

- C:\Program Files\dotnet
- C:\Program Files (x86)\dotnet

Today, customers need one of those two directories in the path in order to get the “dotnet” experience they want, like if they want a 32-bit or 64-bit “dotnet build”.

Note: There will not be a C:\Program Files (x64) directory on Windows Arm64. We are expected to install 64-bit products (Intel and/or Arm based) in C:\Program Files.

On macOS, .NET is installed to one location:

- /usr/local/share/dotnet 

Going forward

There will be one dotnet in the path, just like today. Much like the 32- and 64-bit support we offer on Windows, we’ll offer both Arm64 and x64 builds of .NET and customers can install both or either, and can control which is in the PATH. That will determine if they get an Arm64 or x64 dotnet build. We are not planning on building a .NET version manager that enables switching which architecture you get. We intend to rely on the PATH.

The question is what structure we offer for the two 64-bit products, how intuitive that is (now and later) and how expensive that is for us to offer.

Native dotnet

Premise: there is a “dotnet” directory on every OS, and it is the native architecture.

We’d end up with the following, on Windows and macOS, respectively:

• C:\Program Files\dotnet
• C:\Program Files\dotnet_x64
• /usr/local/share/dotnet
• /usr/local/share/dotnet_x64

This is the usual “who gets the good name” problem.

Pros:

• The dotnet directory is the native architecture on all Oses.
• This option is particularly beneficial for macOS, since we expect x64 emulation to be taken away relatively quickly, at which point dotnet_x64 would disappear on that OS.

Cons:

• x64 directories on x64 and Arm64 machines no longer match.
• x64 installers needs to change, including for 3.1 and 5.0. Hopefully, a single install can install to dotnet on an x64 machine and dotnet_x64 on an Arm64 machine. If not, we need a second set of x64 installers. That’s likely untenable.
• We have to break, migrate or somehow manage existing x64 .NET installs on Apple Silicon machines.

Archify dotnet

Premise: fully embrace multi-arch support, with arch-specific directories.

We’d end up with the following:

• C:\Program Files\dotnet_arm64
• C:\Program Files\dotnet_x64
• /usr/local/share/dotnet _arm64
• /usr/local/share/dotnet _x64

On Arm64, we’d have these arch-specific directories. This is the “no one gets the good name option; everybody loses” option.

Pros:

• The directories are all self-descriptive and symmetrical.

Cons

• Same cons as “Native dotnet”
• Install directories on Windows x64 and Windows Arm64 don’t match.
• We’re stuck with dotnet_arm64 on macOS forever, even though that’s the only architecture supported (in the long run).

Hide architecture differences

Premise: These differences don’t need to be so apparent. We already have version folders under dotnet. We can add arch folders (or something similar). This option has a lot of sub-options, too.

Option 1 – Insert a new folder, with discrete .NET hives underneath:

• C:\Program Files\dotnet\arm64
• C:\Program Files\dotnet\x64
• /usr/local/share/dotnet/arm64
• /usr/local/share/dotnet/x64

Option 2 – Intermix architectures in one structure (here, just shown with Windows, for simplicity):

• For “dotnet”
  • C:\Program Files\dotnet\arm64\dotnet.exe
  • C:\Program Files\dotnet\x64\dotnet.exe
• For frameworks
  • C:\Program Files\dotnet\shared_arm64\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\
  • C:\Program Files\dotnet\shared_x64\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\

Put “x64” and “arm64” somewhere in the folder hierarchy. Where it is, is an implementation decision. It is the same (in spirit) as C:\Windows\Microsoft.NET\Framework and C:\Windows\Microsoft.NET\Framework64.

Option 3 – Arm64 is native architecture with x64 intermixed in:

• For “dotnet”
  • C:\Program Files\dotnet/dotnet
  • C:\Program Files\dotnet/x64/dotnet
• For frameworks
  • C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\
  • C:\Program Files\dotnet\shared_x64\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\

Pros:

• We can manage Arm64 and x64 builds within a single structure in a way that we see as best, both now and over time.

Cons

• Same cons as “Native dotnet”

Hide architectural differences with universal binaries

Both Windows and macOS offer a form of universal binaries, which we could use as a multiplexer between the two 64bit products. There are two problems with that.

• We don’t have enough information in all scenarios to know which architecture is the right one.
• The Windows implementation (per our understanding) isn’t as feature rich as we’d need to make all scenarios work, like debuggers.

We’re rejecting this option for now. We could decide to adopt this solution on macOS and not Windows, should feedback push us in that direction.

Compatibility Wins

Premise: People have already installed .NET x64 on Apple Silicon machines. We need to respect that. This is the opposite of “Native dotnet”

We’d end up with the following:

• C:\Program Files\dotnet_arm64
• C:\Program Files\dotnet
• /usr/local/share/dotnet _arm64
• /usr/local/share/dotnet

Pros:

• Good compatibility for existing Apple Silicon .NET users.

Cons:

• Windows doesn’t necessarily need this solution. We can change .NET before it ships with x64 support.
• x64 gets the “good name” forever.

Hard design questions

• Developer UX
  • Would it be good enough to only have the platform native dotnet available on PATH by default (Native Dotnet solution above)?
  • On Windows x64 the native .NET is typically the one on the PATH is already the case since typically only the x64 dotnet is on PATH, but it’s possible to use it to build, run and test x86 apps.
• User UX
  • can we fully rely on apphost to solve the problem of picking the right archtiecture to run the app with? Different way to ask this: How common is it to run applications via “dotnet app.dll”?
  • Is it possible to release new versions of downlevel installers (3.1 and 5) for macOS x64 and Windows x64 which would be ARM64 aware and could move the default location of the downlevel runtimes?
  • How hard is it to create an x64 installer which can detect that it runs on ARM64 hardware and can change behavior based on that? This is dependent on the installer technology on Windows and macOS.

Other considerations

• We probably need a DOTNET_ROOT(x64), which would be analogous to the existing DOTNET(x86) ENV.
• Installers should be aware of multi-arch OSes and warn users as appropriate

[jkotas]

“Hide architectural differences” option 3

+1

C:\Program Files\dotnet\shared_x64\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\

Why not park the frameworks, sdks and everything else under the x64 directory? ie make this C:\Program Files\dotnet\x64\shared\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\, C:\Program Files\dotnet\x64\sdk\..., etc. The advantage of doing it this way is that everything in x64 environment can assume the same directory structure relative to dotnet.exe.

[jkotas]

It is not unusual that we have to update the shipped .NET versions in servicing to account for OS changes. We have done that number of times. One example from many: https://github.com/dotnet/corefx/pull/34443 . Updating the installers for this (theoretical) case would be one of those changes.