GenAPI issues generate invalid C# for internal types

We have encountered cases where we want to generate internal members in reference assemblies. It would be nice to be able to distinguish between private and internal members instead of the only --all option that accounts for both. However, given the current implementation, here are a few issues we have ran into. FYI @dougbu @wtgodbe @safern @tannergooding

1. Null Argument Exceptions

For example, when generating for the field F:Microsoft.AspNetCore.Mvc.ApplicationModels.ActionAttributeRouteModel.<GetAttributeRoutes>d__3.{}2__current the following exception will be hit

Unhandled exception. System.ArgumentNullException: Value cannot be null. (Parameter 'source')
     at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument)
     at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Func`2 predicate, Boolean& found)
     at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)
     at Microsoft.Cci.Extensions.CSharp.CSharpCciExtensions.GetAttributeOfType(IEnumerable`1 attributes, String attributeName) in /_/src/Microsoft.Cci.Extensions/Extensions/CSharp/CSharpCciExtensions.cs:line 666
     at Microsoft.Cci.Extensions.CSharp.CSharpCciExtensions.GetValueTupleNames(IEnumerable`1 attributes) in /_/src/Microsoft.Cci.Extensions/Extensions/CSharp/CSharpCciExtensions.cs:line 683
     at Microsoft.Cci.Writers.CSharp.CSDeclarationWriter.WriteTypeName(ITypeReference type, Boolean noSpace, IEnumerable`1 attributes, Boolean useTypeKeywords, Boolean omitGenericTypeList) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSDeclarationWriter.cs:line 272
     at Microsoft.Cci.Writers.CSharp.CSDeclarationWriter.WriteFieldDefinition(IFieldDefinition field) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSDeclarationWriter.Fields.cs:line 56
     at Microsoft.Cci.Writers.CSharp.CSDeclarationWriter.WriteMemberDeclaration(ITypeDefinitionMember member) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSDeclarationWriter.cs:line 180
     at Microsoft.Cci.Writers.CSharp.CSDeclarationWriter.WriteDeclaration(IDefinition definition) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSDeclarationWriter.cs:line 120
     at Microsoft.Cci.Writers.CSharpWriter.Visit(ITypeDefinitionMember member) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSharpWriter.cs:line 244
     at Microsoft.Cci.Traversers.SimpleTypeMemberTraverser.Visit(IEnumerable`1 members) in /_/src/Microsoft.Cci.Extensions/Traversers/SimpleTypeMemberTraverser.cs:line 104
     at Microsoft.Cci.Writers.CSharpWriter.Visit(IEnumerable`1 members) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSharpWriter.cs:line 147
     at Microsoft.Cci.Traversers.SimpleTypeMemberTraverser.Visit(ITypeDefinition parentType, IEnumerable`1 fields) in /_/src/Microsoft.Cci.Extensions/Traversers/SimpleTypeMemberTraverser.cs:line 117
     at Microsoft.Cci.Writers.CSharpWriter.Visit(ITypeDefinition parentType, IEnumerable`1 fields) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSharpWriter.cs:line 211
     at Microsoft.Cci.Traversers.SimpleTypeMemberTraverser.Visit(ITypeDefinition type) in /_/src/Microsoft.Cci.Extensions/Traversers/SimpleTypeMemberTraverser.cs:line 74
     at Microsoft.Cci.Writers.CSharpWriter.Visit(ITypeDefinition type) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSharpWriter.cs:line 137
     at Microsoft.Cci.Traversers.SimpleTypeMemberTraverser.Visit(IEnumerable`1 types) in /_/src/Microsoft.Cci.Extensions/Traversers/SimpleTypeMemberTraverser.cs:line 62
     at Microsoft.Cci.Writers.CSharpWriter.Visit(IEnumerable`1 types) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSharpWriter.cs:line 116
     at Microsoft.Cci.Traversers.SimpleTypeMemberTraverser.Visit(ITypeDefinition type) in /_/src/Microsoft.Cci.Extensions/Traversers/SimpleTypeMemberTraverser.cs:line 79
     at Microsoft.Cci.Writers.CSharpWriter.Visit(ITypeDefinition type) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSharpWriter.cs:line 137
     at Microsoft.Cci.Traversers.SimpleTypeMemberTraverser.Visit(IEnumerable`1 types) in /_/src/Microsoft.Cci.Extensions/Traversers/SimpleTypeMemberTraverser.cs:line 62
     at Microsoft.Cci.Writers.CSharpWriter.Visit(IEnumerable`1 types) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSharpWriter.cs:line 116
     at Microsoft.Cci.Traversers.SimpleTypeMemberTraverser.Visit(INamespaceDefinition ns) in /_/src/Microsoft.Cci.Extensions/Traversers/SimpleTypeMemberTraverser.cs:line 55
     at Microsoft.Cci.Writers.CSharpWriter.Visit(INamespaceDefinition ns) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSharpWriter.cs:line 105
     at Microsoft.Cci.Traversers.SimpleTypeMemberTraverser.Visit(IEnumerable`1 namespaces) in /_/src/Microsoft.Cci.Extensions/Traversers/SimpleTypeMemberTraverser.cs:line 43
     at Microsoft.Cci.Traversers.SimpleTypeMemberTraverser.Visit(IAssembly assembly) in /_/src/Microsoft.Cci.Extensions/Traversers/SimpleTypeMemberTraverser.cs:line 35
     at Microsoft.Cci.Writers.CSharpWriter.Visit(IAssembly assembly) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSharpWriter.cs:line 91
     at Microsoft.Cci.Writers.CSharpWriter.WriteAssemblies(IEnumerable`1 assemblies) in /_/src/Microsoft.Cci.Extensions/Writers/CSharp/CSharpWriter.cs:line 79
     at Microsoft.DotNet.GenAPI.Program.<>c__DisplayClass0_0.<Main>b__0() in /_/src/Microsoft.DotNet.GenAPI/Program.cs:line 97
     at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
     at Microsoft.DotNet.GenAPI.Program.Main(String[] args) in /_/src/Microsoft.DotNet.GenAPI/Program.cs:line 165

1. Anonymous classes

Anonymous classes are generated when the --all option is specified and the code is not compilable. We have to manually remove the anonymous classes:

    public static partial class DataProtectionProvider
    {
        public static Microsoft.AspNetCore.DataProtection.IDataProtectionProvider Create(System.IO.DirectoryInfo keyDirectory) { throw null; }
        public static Microsoft.AspNetCore.DataProtection.IDataProtectionProvider Create(System.IO.DirectoryInfo keyDirectory, System.Action<Microsoft.AspNetCore.DataProtection.IDataProtectionBuilder> setupAction) { throw null; }
        public static Microsoft.AspNetCore.DataProtection.IDataProtectionProvider Create(System.IO.DirectoryInfo keyDirectory, System.Action<Microsoft.AspNetCore.DataProtection.IDataProtectionBuilder> setupAction, System.Security.Cryptography.X509Certificates.X509Certificate2 certificate) { throw null; }
        public static Microsoft.AspNetCore.DataProtection.IDataProtectionProvider Create(System.IO.DirectoryInfo keyDirectory, System.Security.Cryptography.X509Certificates.X509Certificate2 certificate) { throw null; }
        public static Microsoft.AspNetCore.DataProtection.IDataProtectionProvider Create(string applicationName) { throw null; }
        public static Microsoft.AspNetCore.DataProtection.IDataProtectionProvider Create(string applicationName, System.Security.Cryptography.X509Certificates.X509Certificate2 certificate) { throw null; }
        internal static Microsoft.AspNetCore.DataProtection.IDataProtectionProvider CreateProvider(System.IO.DirectoryInfo keyDirectory, System.Action<Microsoft.AspNetCore.DataProtection.IDataProtectionBuilder> setupAction, System.Security.Cryptography.X509Certificates.X509Certificate2 certificate) { throw null; }
        [System.Runtime.CompilerServices.CompilerGeneratedAttribute]
        private sealed partial class <>c
        {
            public static readonly Microsoft.AspNetCore.DataProtection.DataProtectionProvider.<>c __9;
            public static System.Action<Microsoft.AspNetCore.DataProtection.IDataProtectionBuilder> __9__1_0;
            public static System.Action<Microsoft.AspNetCore.DataProtection.IDataProtectionBuilder> __9__4_0;
            public <>c() { }
            private static void .cctor() { }
            internal void <Create>b__1_0(Microsoft.AspNetCore.DataProtection.IDataProtectionBuilder builder) { }
            internal void <Create>b__4_0(Microsoft.AspNetCore.DataProtection.IDataProtectionBuilder builder) { }
        }
        [System.Runtime.CompilerServices.CompilerGeneratedAttribute]
        private sealed partial class <>c__DisplayClass0_0
        {
            public string applicationName;
            public <>c__DisplayClass0_0() { }
            internal void <Create>b__0(Microsoft.AspNetCore.DataProtection.IDataProtectionBuilder builder) { }
        }
        [System.Runtime.CompilerServices.CompilerGeneratedAttribute]
        private sealed partial class <>c__DisplayClass3_0
        {
            public string applicationName;
            public <>c__DisplayClass3_0() { }
            internal void <Create>b__0(Microsoft.AspNetCore.DataProtection.IDataProtectionBuilder builder) { }
        }
    }

1. Private static constructors

Non-compilable private static constructor are generated, which we have to remove manually:

    internal static partial class EncodingUtil
    {
        public static readonly System.Text.UTF8Encoding SecureUtf8Encoding;
        private static void .cctor() { }
    }

1. Unsafe interfaces missing the unsafe keyword:

Implementation C# source:

    internal unsafe interface IBCryptGenRandom
    {
        void GenRandom(byte* pbBuffer, uint cbBuffer);
    }

Generated reference C# source

    internal partial interface IBCryptGenRandom
    {
        void GenRandom(byte* pbBuffer, uint cbBuffer);
    }

1. Nullable types not generated

It seems like some nullable types are not generated as such by GenAPI. It doesn’t happen for all nullable types so it’s unclear what exactly causes the errors.

Implementation C# source:

namespace Microsoft.AspNetCore.Mvc.Infrastructure
{
#nullable enable
    internal abstract class ResourceInvoker
    {
        protected readonly DiagnosticListener _diagnosticListener;
        protected readonly ILogger _logger;
        protected readonly IActionContextAccessor _actionContextAccessor;
        protected readonly IActionResultTypeMapper _mapper;
        protected readonly ActionContext _actionContext;
        protected readonly IFilterMetadata[] _filters;
        protected readonly IList<IValueProviderFactory> _valueProviderFactories;

        private AuthorizationFilterContextSealed? _authorizationContext;
        private ResourceExecutingContextSealed? _resourceExecutingContext;
        private ResourceExecutedContextSealed? _resourceExecutedContext;
        private ExceptionContextSealed? _exceptionContext;
        private ResultExecutingContextSealed? _resultExecutingContext;
        private ResultExecutedContextSealed? _resultExecutedContext;

        // Do not make this readonly, it's mutable. We don't want to make a copy.
        // https://blogs.msdn.microsoft.com/ericlippert/2008/05/14/mutating-readonly-structs/
        protected FilterCursor _cursor;
        protected IActionResult? _result;
        protected object? _instance;
...
}

Generated reference C# source

    [System.Runtime.CompilerServices.NullableAttribute((byte)0)]
    [System.Runtime.CompilerServices.NullableContextAttribute((byte)1)]
    internal abstract partial class ResourceInvoker
    {
        protected readonly Microsoft.AspNetCore.Mvc.ActionContext _actionContext;
        protected readonly Microsoft.AspNetCore.Mvc.Infrastructure.IActionContextAccessor _actionContextAccessor;
        [System.Runtime.CompilerServices.NullableAttribute((byte)2)]
        private Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.AuthorizationFilterContextSealed _authorizationContext;
        protected Microsoft.AspNetCore.Mvc.Filters.FilterCursor _cursor;
        protected readonly System.Diagnostics.DiagnosticListener _diagnosticListener;
        [System.Runtime.CompilerServices.NullableAttribute((byte)2)]
        private Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ExceptionContextSealed _exceptionContext;
        protected readonly Microsoft.AspNetCore.Mvc.Filters.IFilterMetadata[] _filters;
        [System.Runtime.CompilerServices.NullableAttribute((byte)2)]
        protected object _instance;
        protected readonly Microsoft.Extensions.Logging.ILogger _logger;
        protected readonly Microsoft.AspNetCore.Mvc.Infrastructure.IActionResultTypeMapper _mapper;
        [System.Runtime.CompilerServices.NullableAttribute((byte)2)]
        private Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResourceExecutedContextSealed _resourceExecutedContext;
        [System.Runtime.CompilerServices.NullableAttribute((byte)2)]
        private Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResourceExecutingContextSealed _resourceExecutingContext;
        [System.Runtime.CompilerServices.NullableAttribute((byte)2)]
        protected Microsoft.AspNetCore.Mvc.IActionResult _result;
        [System.Runtime.CompilerServices.NullableAttribute((byte)2)]
        private Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultExecutedContextSealed _resultExecutedContext;
        [System.Runtime.CompilerServices.NullableAttribute((byte)2)]
        private Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultExecutingContextSealed _resultExecutingContext;
        protected readonly System.Collections.Generic.IList<Microsoft.AspNetCore.Mvc.ModelBinding.IValueProviderFactory> _valueProviderFactories;
...
}

In this case, it looks like NullableAttribute is added to the nullable types but the types are not nullable themselves. Is this an oversight or a feature that we are misunderstanding?

To repro these issues, find the implementation projects containing the code I’ve pasted here and run dotnet msbuild /t:GenerateReferenceSource to run genAPI to produce the generated code.

[ericstj]

For Nullable this was all a recent addition made by @safern. You are probably using an old version of GenAPI.

@dougbu it’s reasonable to have a new filter that is public + internals-only-when-IVT is used. This shouldn’t be the default public filter though since that would break folks who use GenAPI and only use IVT for runtime assemblies and testing.

[dougbu]

All but “Nullable types not generated” addressed in d6dd7a000fe9. Already have a number of issues related to nullability elsewhere in this repo and that point feels redundant.