RTD async functions freeze excel

Hi there,

I’ve been working with Excel DNA to produce an excel plugin that predominantly interfaces with my company’s web API (via a C# client that just wraps calls via an HTTP client) to provide modelled financial data (generally with the end goal of plotting the returned data series on charts).

In general, we’ve had a lot of success. However, today I ran into a scenario where I managed to lock up Excel by making successive requests to a particular async function.

Assuming the following changes in my spreadsheet:

Step 1

	A	B
1	13-Aug-19	=my_long_running_func(A1)

Step 2

	A	B
1	09-Aug-19	=my_long_running_func(A1)

Step 3

	A	B
1	13-Aug-19	=my_long_running_func(A1)

my_long_running_func receives 3 calls. To the first two, it returns #N/A as expected, but the third never returns after making the API call. Excel locks up and control is only returned when the API call times out several minutes later, with an exception: A task was canceled..

While the long running API call and probably the exception thrown are both nothing to do with Excel DNA and things that we need to fix too, I was wondering if you’d be able to suggest any reason why Excel would lock up despite us running these calls as async functions. The common theme I can see is the parameter change back to a value the function has already seen (13-Aug-19) while the async function is still in flight.

my_long_running_func roughly looks like:

        [ExcelFunction(Name = "my_long_running_func", Description = "Runs for a long time")]
        public static object LongRunningFunc(
            [ExcelArgument(Name = "Date")] DateTime date)
        {
            var parameters = new object[] { date };
            return ExcelAsyncUtils.Run(
                "LongRunningFunc",
                parameters,
                delegate
                {
                    // ApiRequest is an object provided by the API client
                    var request = new ApiRequest
                                  {
                                      Date = date,
                                  };
                    var response = GetApiClient().CalculateModel(request).GetAwaiter().GetResult();
                    // Parse response and put it into an object[,]; the code never reaches this point
                    return output;
                });
        }

We have also tried using https://github.com/StephenCleary/AsyncEx which has helped us out in a few scenarios where the .GetAwaiter().GetResult() pattern sent us into deadlock. But switching var response ... to var response = AsyncContext.Run(() => GetApiClient().CalculateModel(request)) does not change the behaviour described.

I’ve been looking through ExcelDna.Integration/ExcelRtdObserver.cs as I was wondering if your caching mechanism was waiting on the result of the async function (which would mean further calls with matching parameters could potentially be blocked(?)), but I can’t see any evidence of that.

Apologies, I’m aware of how vague a problem this is, but I was hoping that someone with more expertise might see an error in my use of the library or might be able to provide an explanation on how the function ultimately gets called, and in turn provide some clues as to why our API client is blocking the main thread.

Any input would be hugely useful as the library has, thus far, worked marvellously.

[govert]

@andrewbridge You’re seeing the default behaviour of the .NET ThreadPool when Tasks are enqueued. Basically the ThreadPool starts small, and with your current code (basically the default Excel-DNA async API) your slow Tasks are using up the initially small ThreadPool quickly - every outstanding delegate uses up a ThreadPool thread until it is complete.

A simple test to confirm this is to make the initial ThreadPool large by adding a call like this when your add-in is loaded:

    System.Threading.ThreadPool.SetMinThreads(200, 200);

This will make sure your Tasks enqueue quickly (thus making the outgoing calls quickly). But that’s not the optimal use of the ThreadPool or the .NET Task-based infrastructure .

The better approach is to use the Task-based async/await throughout - using async Tasks as supported by the HttpClient class should not use a ThreadPool except possibly for the ‘fast’ synchronous completion code. To do this means you need to replace ExcelAsyncUtil.Run with another implementation that is a Task-based wrapper on top of IExcelObservable.

I have an extension library called [ExcelDna.Registration](https://github.com/Excel-DNA/Registration) that does all of this automatically (or has some code like AsyncTaskUtil.cs to help you implement it yourself). This means you can change your exported function from the HangingExcel sample to look like this:

        [ExcelFunction(Name = "hangexcel_long", Description = "Do a long async call")]
        public static async Task<object> LongRun(string name, DateTime when)
        {
            var request = new DetailsRequest { Name = name, Date = when };
            var client = new FakeCallsClient();
            // Vary long calls
            var response = await (longCallCount % 2 == 0 ? client.GetColoursTooSlowlyAsync(request) : client.GetColoursVerySlowlyAsync(request));
            longCallCount++;
            return "Hello " + name + " on " + when.ToShortDateString();
        }

Note that is is an async / await function returning Task<T> directly. The magic is in the Registration library, which is called in an AutoOpen helper:

    public class AddIn : IExcelAddIn
    {
        public void AutoOpen()
        {
            ExcelRegistration.GetExcelFunctions()
                             .ProcessAsyncRegistrations(nativeAsyncIfAvailable: false)
                             .RegisterFunctions();
        }

        public void AutoClose()
        {
        }
    }

I’ve made a pull request against your test project that should compile and give you the parallel requests you expect.

[govert]

Putting TryGetWorkbookReference on main thread…is that possible?

You just have to call it directly from the ribbon callback, before you start the async work.