Race condition in WorkScheduler

I had just deployed a new beta of my app to a device and went to show my PM, I thought I had fixed all the known issues and I had tested well before. On the way to the PM the WiFi signal dropped and the phone went over to another network which needed credentials to be entered on a webpage for any traffic to let through.

My PM opened the App and 💥 crash! It had been opened just before I went to see my PM, so no more than a couple of minutes had passed, so the app didn’t re-load but was in the same state as I left it.

Anyways, something is not right and here is the crash log I have from HockeyApp:

Xamarin caused by: android.runtime.JavaProxyThrowable: System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. ---> System.InvalidOperationException: Cannot call Dequeue() on an empty queue
  at FFImageLoading.Concurrency.SimplePriorityQueue`2[TItem,TPriority].Dequeue () [0x0001f] in C:\projects\ffimageloading\source\FFImageLoading.Common\Concurrency\FastPriorityQueue\SimplePriorityQueue.cs:104 
  at FFImageLoading.Work.WorkScheduler+<TakeFromPendingTasksAndRunAsync>d__49.MoveNext () [0x0015b] in C:\projects\ffimageloading\source\FFImageLoading.Common\Work\WorkScheduler.cs:311 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <3fd174ff54b146228c505f23cf75ce71>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <3fd174ff54b146228c505f23cf75ce71>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <3fd174ff54b146228c505f23cf75ce71>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <3fd174ff54b146228c505f23cf75ce71>:0 
  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult () [0x00000] in <3fd174ff54b146228c505f23cf75ce71>:0 
  at FFImageLoading.Work.WorkScheduler+<<TakeFromPendingTasksAndRun>b__47_0>d.MoveNext () [0x00024] in C:\projects\ffimageloading\source\FFImageLoading.Common\Work\WorkScheduler.cs:252 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.InvalidOperationException: Cannot call Dequeue() on an empty queue
  at FFImageLoading.Concurrency.SimplePriorityQueue`2[TItem,TPriority].Dequeue () [0x0001f] in C:\projects\ffimageloading\source\FFImageLoading.Common\Concurrency\FastPriorityQueue\SimplePriorityQueue.cs:104 
  at FFImageLoading.Work.WorkScheduler+<TakeFromPendingTasksAndRunAsync>d__49.MoveNext () [0x0015b] in C:\projects\ffimageloading\source\FFImageLoading.Common\Work\WorkScheduler.cs:311 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <3fd174ff54b146228c505f23cf75ce71>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <3fd174ff54b146228c505f23cf75ce71>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <3fd174ff54b146228c505f23cf75ce71>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <3fd174ff54b146228c505f23cf75ce71>:0 
  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult () [0x00000] in <3fd174ff54b146228c505f23cf75ce71>:0 
  at FFImageLoading.Work.WorkScheduler+<<TakeFromPendingTasksAndRun>b__47_0>d.MoveNext () [0x00024] in C:\projects\ffimageloading\source\FFImageLoading.Common\Work\WorkScheduler.cs:252 <---

Looking at the code where the exception originates from: https://github.com/luberda-molinet/FFImageLoading/blob/683ec66828b178d55ce51fe2887204aefcd7cfe9/source/FFImageLoading.Common/Work/WorkScheduler.cs#L304

while (tasksToRun.Count < numberOfTasks && PendingTasks.Count > 0)
{
    var task = PendingTasks.Dequeue();

The Dequeue call claims that PendingTasks.Count is 0 but you just checked right before that it wasn’t. This sounds like a race condition to me, something else must have done something to the PendingTasks to become 0.

Another problem is, that I cannot catch this exception, it bubbles up right until: https://github.com/luberda-molinet/FFImageLoading/blob/683ec66828b178d55ce51fe2887204aefcd7cfe9/source/FFImageLoading.Common/Work/WorkScheduler.cs#L241

Here you do:

protected void TakeFromPendingTasksAndRun()
{
    Task.Factory.StartNew(async () =>
    {
        await TakeFromPendingTasksAndRunAsync().ConfigureAwait(false); // FMT: we limit concurrent work using MaxParallelTasks
    }, CancellationToken.None, TaskCreationOptions.DenyChildAttach | TaskCreationOptions.HideScheduler, TaskScheduler.Default).ConfigureAwait(false);
}

Since the Task is just wrapped in StartNew you are essentially doing async void. If something throws here, the entire App explodes. Please wrap this in a try/catch at least.

[rodhemphill]

Thanks guys. @Cheesebaron - I (now) understand mine is not the same issue - I’m happy to raise a different issue. @daniel-luberda Correct these are only logs and the app doesn’t crash - which means they must be trapped within the app somewhere (it’s not my app - I’ll -investigation this further). If I comment out the ffimageloading from the XAML the logs disappear plus they mention FFImageLoading continuously (there are thousands of lines of logs). The rest of the code does a “CanConnect” call before it attempts to access the network which fails at the time this error occurs. I have found your customer logger documentation so I’ll implement that and if there is still a problem I’ll log a different issue. Thanks for you help.

[Cheesebaron]

@rodhemphill you will see TaskAwaiter in your stack traces as soon as you use async/await… So tying those two together doesn’t automatically warrant a link to this issue. Yours is not related to this issue as the stack trace is entirely different. I clearly throws due to a Socket exception, which triggers another part of the code.

This issue is about a race condition which has been fixed.