GPU Interop on Windows, Android, MacOS, iOS

[PhilippeMonteil]

GPU Interop on Windows, Android, MacOS, iOS

I am trying to find a way to inject GPU renderings in a Skia rendered Avalonia interface
on Windows, Android, MacOS and iOS.

I have explored two approaches, one based on a Composition and the other on Custom Draw Operations.

They seem to be promising on Windows, Android, MacOS but to be both deadlocks on iOS as briefly described below.

Composition

Based on the Avalonia/GpuInterop project DrawingSurfaceDemoBase.cs module,
I have tried to obtain the necessary ICompositionGpuInterop interop like this :

    protected override async void OnAttachedToVisualTree(VisualTreeAttachmentEventArgs e)
    {
        base.OnAttachedToVisualTree(e);
        try
        {
            compositor = ElementComposition.GetElementVisual(this)!.Compositor;

            Debug.WriteLine($"{GetType().Name}.{nameof(OnAttachedToVisualTree)}(-) 1.200");

            {
                interop = await compositor.TryGetCompositionGpuInterop();
                Debug.WriteLine($"interop={interop}");

                if (interop != null)
                {
                    Debug.WriteLine($"interop.SupportedImageHandleTypes(-)=");
                    foreach (string _s in interop.SupportedImageHandleTypes)
                    {
                        Debug.WriteLine($"    '{_s}'");
                    }
                    Debug.WriteLine($"interop.SupportedImageHandleTypes(+)");
                }
            }

I obtain the following debug on the various platforms :

// Windows
// DrawingSurfaceDemoBase.OnAttachedToVisualTree(-) 1.200
// interop=Avalonia.Rendering.Composition.CompositionInterop
//     interop.SupportedImageHandleTypes(-)=
//     'D3D11TextureGlobalSharedHandle'
//     'D3D11TextureNtHandle'
// interop.SupportedImageHandleTypes(+)

// Android
// DrawingSurfaceDemoBase.OnAttachedToVisualTree(-) 1.200
// interop=Avalonia.Rendering.Composition.CompositionInterop
//     interop.SupportedImageHandleTypes(-)=
// interop.SupportedImageHandleTypes(+)
//     contextSharingFeature=Avalonia.Skia.GlSkiaGpu

// MacOS
// DrawingSurfaceDemoBase.OnAttachedToVisualTree(-) 1.200
// interop=Avalonia.Rendering.Composition.CompositionInterop
//     interop.SupportedImageHandleTypes(-)=
//     'IOSurfaceRef'
// interop.SupportedImageHandleTypes(+)
        
// iOS
// DrawingSurfaceDemoBase.OnAttachedToVisualTree(-) 1.200
// interop=

That is : a null on iOS which seems to make the approach impossible on that platform.

Custom Draw Operations

public class ChartControl : Control
{
    public override void Render(DrawingContext context)
    {
        context.Custom(new ChartDrawOperation(new Rect(Bounds.Size)));
    }

    private class ChartDrawOperation : ICustomDrawOperation
    {
    
        public void Render(ImmediateDrawingContext context)
        {
            var feature = context.TryGetFeature<ISkiaSharpApiLeaseFeature>();
            if (feature is null) return;

            using ISkiaSharpApiLease lease = feature.Lease();

            Debug.WriteLine($"{GetType().Name}.Render 1.900 lease={lease}");
            Debug.WriteLine($"    .GrContext={lease.GrContext}");
            Debug.WriteLine($"    .SkSurface={lease.SkSurface}");

            using (ISkiaSharpPlatformGraphicsApiLease _pISkiaSharpPlatformGraphicsApiLease = lease.TryLeasePlatformGraphicsApi())
            {
                Debug.WriteLine($"    _pISkiaSharpPlatformGraphicsApiLease={_pISkiaSharpPlatformGraphicsApiLease}");
                IPlatformGraphicsContext _pIPlatformGraphicsContext = _pISkiaSharpPlatformGraphicsApiLease?.Context;
                Debug.WriteLine($"    _pIPlatformGraphicsContext={_pIPlatformGraphicsContext}");
            }

    }

I obtain the following debug on the various platforms :

    // Windows :
    // ChartDrawOperation.Render 1.100 lease=Avalonia.Skia.DrawingContextImpl+SkiaLeaseFeature+ApiLease
    //     .GrContext=SkiaSharp.GRContext
    //     .SkSurface=SkiaSharp.SKSurface
    //     _pISkiaSharpPlatformGraphicsApiLease=Avalonia.Skia.DrawingContextImpl+SkiaLeaseFeature+ApiLease+PlatformApiLease
    // _pIPlatformGraphicsContext=Avalonia.OpenGL.Egl.EglContext

    // Android :
    // ChartDrawOperation.Render 1.100 lease=Avalonia.Skia.DrawingContextImpl+SkiaLeaseFeature+ApiLease
    //     .GrContext=SkiaSharp.GRContext
    //     .SkSurface=SkiaSharp.SKSurface
    //     _pISkiaSharpPlatformGraphicsApiLease=Avalonia.Skia.DrawingContextImpl+SkiaLeaseFeature+ApiLease+PlatformApiLease
    // _pIPlatformGraphicsContext=Avalonia.OpenGL.Egl.EglContext

    // MacOS :
    // ChartDrawOperation.Render 1.100 lease=Avalonia.Skia.DrawingContextImpl+SkiaLeaseFeature+ApiLease
    //     .GrContext=SkiaSharp.GRContext
    //     .SkSurface=SkiaSharp.SKSurface
    //     _pISkiaSharpPlatformGraphicsApiLease=Avalonia.Skia.DrawingContextImpl+SkiaLeaseFeature+ApiLease+PlatformApiLease
    // _pIPlatformGraphicsContext=Avalonia.Native.GlContext
        
    // iOS :
    // ChartDrawOperation.Render 1.100 lease=Avalonia.Skia.DrawingContextImpl+SkiaLeaseFeature+ApiLease
    //     .GrContext=SkiaSharp.GRContext
    //     .SkSurface=SkiaSharp.SKSurface
    //     _pISkiaSharpPlatformGraphicsApiLease=Avalonia.Skia.DrawingContextImpl+SkiaLeaseFeature+ApiLease+PlatformApiLease
    // _pIPlatformGraphicsContext=Avalonia.iOS.Metal.MetalDevice

The Windows, Android and MacOS platforms seem to give access to some IGlContext interface.

namespace Avalonia.OpenGL.Egl
{
    public class EglContext : IGlContext
    {
        ...
    }

    public interface IGlContext : IPlatformGraphicsContext
    {
        GlVersion Version { get; }
        GlInterface GlInterface { get; }
        int SampleCount { get; }
        int StencilSize { get; }
        IDisposable MakeCurrent();
        bool IsSharedWith(IGlContext context);
        bool CanCreateSharedContext { get; }
        IGlContext? CreateSharedContext(IEnumerable<GlVersion>? preferredVersions = null);
    }

namespace Avalonia.OpenGL
{
    
    class GlContext : IGlContext
    {
        ...
    }

The iOS platform seem to give access to an instance of the MetalDevice private class,
exposing the Avalonia.iOS.Metal.MetalDevice public interface :

namespace Avalonia.Native;

class MetalDevice : IMetalDevice
{

IMetalDevice exposes a pair of .Device / .CommandQueue members which might give access to the underlying
Metal Device and CommandQueue.

Those members are private and I couldn't find a way to access them ...

namespace Avalonia.Metal;

public interface IMetalDevice : IPlatformGraphicsContext
{
    IntPtr Device { get; }
    IntPtr CommandQueue { get; }
}