[SPEC] External GPU memory interop (OpenGL, Vulkan, DirectX)

To avoid jittering and rendering artifacts, any user-rendered GPU images need to be properly synchronized. By properly synchronized we mean:

• user rendering commands are already completed when Avalonia tries to consume the rendered contents
• rendered contents are synchronized with the rest of the changes to the visual and composition trees, so they should be applied with the rest of the composition batch
• rendering should be synchronized with monitor refresh rate

We are adding CompositionDrawingSurface and CompositionSurfaceVisual APIs (CompositionBrush and CompositionSpriteVisual would come later).

User code could use ElementComposition.SetElementChildVisual to display the CompositionSurfaceVisual on top of their control.

public class CompositionSurface : CompositionObject
{
}

public class CompositionSurfaceVisual : CompositionVisual
{
    public CompositionSurface { get; set; }
}


public class Compositor
{
...
    public CompositionDrawingSurface CreateDrawingSurface();
    public CompositionSurfaceVisual CreateSurfaceVisual();
...
}

CompositionDrawingSurface retains its own copy of the image. To update said copy one needs to provide a GPU-backed image with some means to wait for rendering to be completed:

public class CompositionDrawingSurface : CompositionSurface
{
    /// <summary>
    /// Updates the surface contents using an imported memory image using a keyed mutex as the means of synchronization
    /// </summary>
    /// <param name="image">GPU image with new surface contents</param>
    /// <param name="acquireIndex">The mutex key to wait for before accessing the image</param>
    /// <param name="releaseIndex">The mutex key to release for after accessing the image </param>
    /// <returns>A task that completes when update operation is completed and user code is free to destroy or dispose the image</returns>
    public Task UpdateWithKeyedMutex(ICompositionImportedGpuImage image, uint acquireIndex, uint releaseIndex);

    /// <summary>
    /// Updates the surface contents using an imported memory image using a semaphore pair as the means of synchronization
    /// </summary>
    /// <param name="image">GPU image with new surface contents</param>
    /// <param name="waitForSemaphore">The semaphore to wait for before accessing the image</param>
    /// <param name="signalSemaphore">The semaphore to signal after accessing the image</param>
    /// <returns>A task that completes when update operation is completed and user code is free to destroy or dispose the image</returns>
    public Task UpdateWithSemaphores(ICompositionImportedGpuImage image,
        ICompositionImportedGpuSemaphore waitForSemaphore,
        ICompositionImportedGpuSemaphore signalSemaphore);

    /// <summary>
    /// Updates the surface contents using an unspecified automatic means of synchronization
    /// provided by the underlying platform
    /// </summary>
    /// <param name="image">GPU image with new surface contents</param>
    /// <returns>A task that completes when update operation is completed and user code is free to destroy or dispose the image</returns>
    public Task UpdateWithAutomaticSync(ICompositionImportedGpuImage image);
}

What is ICompositionImportedGpuImage? It’s an object that represents an externally allocated GPU memory, that’s been imported to use with the compositor and the current GPU context (it will become invalid once that context is lost).

public class Compositor
{
...
    public ValueTask<ICompositionGpuInterop?> TryGetGpuInterop();
...
}


public interface ICompositionGpuInterop
{
    /// <summary>
    /// Returns the list of image handle types supported by the current GPU backend, see <see cref="KnownPlatformGraphicsExternalImageHandleTypes"/>
    /// </summary>
    IReadOnlyList<string> SupportedImageHandleTypes { get; }
    
    /// <summary>
    /// Returns the list of semaphore types supported by the current GPU backend, see <see cref="KnownPlatformGraphicsExternalSemaphoreTypes"/>
    /// </summary>
    IReadOnlyList<string> SupportedSemaphoreTypes { get; }

    /// <summary>
    /// Returns the supported ways to synchronize access to the imported GPU image
    /// </summary>
    /// <returns></returns>
    CompositionGpuImportedImageSynchronizationCapabilities GetSynchronizationCapabilities(string imageHandleType);
    
    /// <summary>
    /// Asynchronously imports a texture. The returned object is immediately usable.
    /// </summary>
    ICompositionImportedGpuImage ImportImage(IPlatformHandle handle,
        PlatformGraphicsExternalMemoryProperties properties);

    /// <summary>
    /// Asynchronously imports a texture. The returned object is immediately usable.
    /// </summary>
    /// <param name="image">An image that belongs to the same GPU context or the same GPU context sharing group as one used by compositor</param>
    ICompositionImportedGpuImage ImportImage(ICompositionImportableSharedGpuContextImage image);

    /// <summary>
    /// Asynchronously imports a semaphore object. The returned object is immediately usable.
    /// </summary>
    ICompositionImportedGpuSemaphore ImportSemaphore(IPlatformHandle handle);
    
    /// <summary>
    /// Asynchronously imports a semaphore object. The returned object is immediately usable.
    /// </summary>
    /// <param name="image">A semaphore that belongs to the same GPU context or the same GPU context sharing group as one used by compositor</param>
    ICompositionImportedGpuImage ImportSemaphore(ICompositionImportableSharedGpuContextSemaphore image);
    
    /// <summary>
    /// Indicates if the device context this instance is associated with is no longer available
    /// </summary>
    public bool IsLost { get; }
    
}

[Flags]
public enum CompositionGpuImportedImageSynchronizationCapabilities
{
    /// <summary>
    /// Pre-render and after-render semaphores must be provided alongside with the image
    /// </summary>
    Semaphores = 1,
    /// <summary>
    /// Image must be created with D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX or in other compatible way
    /// </summary>
    KeyedMutex = 2,
    /// <summary>
    /// Synchronization and ordering is somehow handled by the underlying platform
    /// </summary>
    Automatic = 4
}

/// <summary>
/// An imported GPU object that's usable by composition APIs 
/// </summary>
public interface ICompositionGpuImportedObject : IDisposable
{
    /// <summary>
    /// Tracks the import status of the object. Once the task is completed,
    /// the user code is allowed to free the resource owner in case when a non-owning
    /// sharing handle was used
    /// </summary>
    Task ImportCompeted { get; }
    /// <summary>
    /// Indicates if the device context this instance is associated with is no longer available
    /// </summary>
    bool IsLost { get; }
}

/// <summary>
/// An imported GPU image object that's usable by composition APIs 
/// </summary>
[NotClientImplementable]
public interface ICompositionImportedGpuImage : ICompositionGpuImportedObject
{

}

/// <summary>
/// An imported GPU semaphore object that's usable by composition APIs 
/// </summary>
[NotClientImplementable]
public interface ICompositionImportedGpuSemaphore : ICompositionGpuImportedObject
{

}

/// <summary>
/// An GPU object descriptor obtained from a context from the same share group as one used by the compositor
/// </summary>
[NotClientImplementable]
public interface ICompositionImportableSharedGpuContextObject : IDisposable
{
}

/// <summary>
/// An GPU image descriptor obtained from a context from the same share group as one used by the compositor
/// </summary>
[NotClientImplementable]
public interface ICompositionImportableSharedGpuContextImage : IDisposable
{
}

/// <summary>
/// An GPU semaphore descriptor obtained from a context from the same share group as one used by the compositor
/// </summary>
[NotClientImplementable]
public interface ICompositionImportableSharedGpuContextSemaphore : IDisposable
{
}

public struct PlatformGraphicsExternalMemoryProperties
{
    public int Width { get; set; }
    public int Height { get; set; }
    public PlatformGraphicsExternalMemoryFormat Format { get; set; }
}

public enum PlatformGraphicsExternalMemoryFormat
{
    R8G8B8A8UNorm,
    B8G8R8A8UNorm
}

/// <summary>
/// Describes various GPU memory handle types that are currently supported by Avalonia graphics backends
/// </summary>
public static class KnownPlatformGraphicsExternalImageHandleTypes
{
    /// <summary>
    /// An DXGI global shared handle returned by IDXGIResource::GetSharedHandle D3D11_RESOURCE_MISC_SHARED or D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag.
    /// The handle does not own the reference to the underlying video memory, so the provider should make sure that the resource is valid until
    /// the handle has been successfully imported
    /// </summary>
    public const string D3D11TextureGlobalSharedHandle = nameof(D3D11TextureGlobalSharedHandle);
    /// <summary>
    /// A DXGI NT handle returned by IDXGIResource1::CreateSharedHandle for a texture created with D3D11_RESOURCE_MISC_SHARED_NTHANDLE or flag
    /// </summary>
    public const string D3D11TextureNtHandle = nameof(D3D11TextureNtHandle);
    /// <summary>
    /// A POSIX file descriptor that's exported by Vulkan using VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT or in a compatible way
    /// </summary>
    public const string VulkanOpaquePosixFileDescriptor = nameof(VulkanOpaquePosixFileDescriptor);
}

/// <summary>
/// Describes various GPU semaphore handle types that are currently supported by Avalonia graphics backends
/// </summary>
public static class KnownPlatformGraphicsExternalSemaphoreTypes
{
    /// <summary>
    /// A POSIX file descriptor that's been exported by Vulkan using VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_FD_BIT or in a compatible way
    /// </summary>
    public const string VulkanOpaquePosixFileDescriptor = nameof(VulkanOpaquePosixFileDescriptor);
    
    /// <summary>
    /// A NT handle that's been exported by Vulkan using VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32_BIT or in a compatible way
    /// </summary>
    public const string VulkanOpaqueNtHandle = nameof(VulkanOpaqueNtHandle);
    
    // A global shared handle that's been exported by Vulkan using VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT or in a compatible way
    public const string VulkanOpaqueKmtHandle = nameof(VulkanOpaqueKmtHandle);
    
    /// A DXGI NT handle returned by ID3D12Device::CreateSharedHandle or ID3D11Fence::CreateSharedHandle
    public const string Direct3D12FenceNtHandle = nameof(Direct3D12FenceNtHandle);
}

ICompositionImportableSharedGpuContextObject and friends would be obtained from IGlContext and are required to support platforms where we provide OpenGL rendering support via OpenGL context sharing and for those who wish to reuse the same VkDevice for both Avalonia and user code.

OpenGlControlBase, VulkanControlBase and swapchains are outside of the scope of this spec. We are getting complaints about our built-in base controls being too inflexible for various applications, so the goal of this spec is to define a flexible API that can be used to implement both our built-in simple controls an complex user scenarios.

For vsync-synchronized rendering, we’ll just add RequestAnimationFrame API that would allow one to do rendering just before the composition batch is sent to the render thread:

public class Compositor
{
...
    public void RequestAnimationFrame(Action<object> callback, object state);
...
}

[kekekeks]

and not a list of strings.

We are using strings to describe IPlatformHandle types in other parts of the framework. That way APIs can be extended from 3rd-party backends if needed.

because this handle type is not Vulkan specific

According to the Vulkan spec: it “specifies a POSIX file descriptor handle that has only limited valid usage outside of Vulkan and other compatible APIs”. This fd can only be exported from Vulkan, it can’t be created by OpenGL. A universal API-agnostic (and usable by multiple physical devices) handle would be the DMABUF one. We’ll add support for those extra handle types at some point too.

Before Vulkan existed, they can be used to share a DirectX 9 or DirectX 11 textures

DirectX-compatible DXGI share handles are represented by VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_BIT and VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_KMT_BIT respectively.

Vulkan has its own VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT and VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT handle types that are not compatible with DirectX, but can be consumed from OpenGL via external objects extension just like on Linux.

[kekekeks]

DirectX 11 supports semaphores via ID3D11Device5::CreateFence / ID3D11Device5::OpenSharedFence. In vulkan those are represented by VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_D3D12_FENCE_BIT.

VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32_BIT and VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_OPAQUE_WIN32_KMT_BIT are Vulkan/OpenGL specific