[dotnet] Unit tests with Helm.V3.Chart throw NullReferenceException

I’m trying to write some unit tests in C# for my stacks and I’m having trouble with a stack which contains a Helm.V3.Chart. Running unit tests which utilize charts currently throw System.NullReferenceException.

Expected behavior

Unit tests execute without exceptions.

Current behavior

The following exception is thrown when executing a unit test with a stack containing Helm.V3.Chart.

   Pulumi.RunException : Running program '<path>\bin\Debug\net5.0\testhost.dll' failed with an unhandled exception:
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Collections.Immutable.ImmutableArray.CreateRange[TSource,TResult](ImmutableArray`1 items, Func`2 selector)
   at Pulumi.Extensions.SelectAsArray[TItem,TResult](ImmutableArray`1 items, Func`2 map)
   at Pulumi.InputList`1.op_Implicit(ImmutableArray`1 values)
   at Pulumi.Kubernetes.Helm.V3.Chart.<>c__DisplayClass3_0.<ParseTemplate>b__0(ImmutableArray`1 objs)
   at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
   at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
   at Pulumi.Output`1.Pulumi.IOutput.GetDataAsync()
   at Pulumi.Serialization.Serializer.SerializeAsync(String ctx, Object prop, Boolean keepResources)
   at Pulumi.Deployment.SerializeFilteredPropertiesAsync(String label, IDictionary`2 args, Predicate`1 acceptKey, Boolean keepResources)
   at Pulumi.Deployment.SerializeAllPropertiesAsync(String label, IDictionary`2 args, Boolean keepResources)
   at Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output`1 outputs)
   at Pulumi.Deployment.Runner.<>c__DisplayClass9_0.<<WhileRunningAsync>g__HandleCompletion|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Pulumi.Deployment.Runner.WhileRunningAsync()
    Stack Trace:
       at Pulumi.Deployment.TestAsync(IMocks mocks, Func`2 runAsync, TestOptions options)
    <snip>
--- End of stack trace from previous location ---

I did some digging and found that the trouble is caused by the implicit conversion operator invoked at https://github.com/pulumi/pulumi-kubernetes/blob/80d4c16c2240d77ca575273a9c167c739e7681a3/sdk/dotnet/Helm/V3/Chart.cs#L365

The actual problem seems to be that the ImmutableArray-parameter https://github.com/pulumi/pulumi-kubernetes/blob/80d4c16c2240d77ca575273a9c167c739e7681a3/sdk/dotnet/Helm/V3/Invokes.cs#L46 given to HelmTemplateResult is uninitialized when running the tests. Since the ImmutableArray is uninitialized, the System.NullReferenceException is thrown when trying to do the implicit conversion.

Steps to reproduce

The problem can be produced by having the following stack:

public class FailingHelmStack : Pulumi.Stack
{
    public FailingHelmStack()
    {
        var chart = new Pulumi.Kubernetes.Helm.V3.Chart("chart"
            , new Pulumi.Kubernetes.Helm.ChartArgs()
            {
                Chart = "ingress-nginx",
                Namespace = "kube-system",
                FetchOptions = new Pulumi.Kubernetes.Helm.ChartFetchArgs
                {
                    Repo = "https://kubernetes.github.io/ingress-nginx",
                },
            });
    }
}

And invoking it with Deployment.TestAsync in a C# test project.

Context (Environment)

Trying to write unit tests for stacks which contain Helm-charts.

Proposed fix

Ensure that the HelmTemplateResult.Result array is always initialized.

~~For example by adding the following to HelmTemplateResult constructor https://github.com/pulumi/pulumi-kubernetes/blob/80d4c16c2240d77ca575273a9c167c739e7681a3/sdk/dotnet/Helm/V3/Invokes.cs#L48~~

Result = result;
// Ensure the Result array is always initialized
if (Result.IsDefault)
{
    Result = ImmutableArray<ImmutableDictionary<string, object>>.Empty;
}

If this seems like a reasonable solution, I can submit a PR.

NOTE I am not sure if similiar problem occurs anywhere else with ImmutableArray

UPDATE The implicit conversion fix is in pulumi/pulumi#6544

[t0yv0]

I have a workaround for your unit test that makes it pass for me:

https://github.com/vipentti/pulumi-helm-chart-unittest-issue/pull/1

At a high level, what I think is happening, the Mock implementation you were using was violating some assumptions of how kubernetes:helm:template resource responds. I could only find this by perusing the provider.go source code in pulumi-kubernetes… but it seems like a “result” property is populated with output to decodeYaml, which has this:

// decodeYaml parses a YAML string, and then returns a slice of untyped structs that can be marshalled into
// Pulumi RPC calls. If a default namespace is specified, set that on the relevant decoded objects.

So mocking the helm resources deeply and accurately sounds like something off the beaten path. I am a bit new to Pulumi and will ask around if there is a better way. Please let me know if my PR unblocks you to get to something sufficiently useful, or you run into more roadblocks trying to mock the Helm provider?

Considering changes to pulumi-kubernetes. Technically with the original shallow mock, this is what’s happening:

V3/Chart.cs calls Invokes.HelmTemplate(..)
which calls Deployement.Instance.InvokeAsync<HelmTemplateResult>("kubernetes:helm:template", .., ..) 
which calls  internal interface IDeployment { Task<T> InvokeAsync(..) } 
which returns null
which gets wrapped in HelmTemplateResult and continues downstream

So we could consider HelmTemplateResult normalizing null to empty ImmutableArray. It is clear though this only affects the scenario of testing with incomplete mocks, and I am wondering if this would actually unblock some useful testing?

[vipentti]

@t0yv0 I’ve created a small repo which displays the problem https://github.com/vipentti/pulumi-helm-chart-unittest-issue