CosmosException and CosmosOperationCanceledException do not include calling code in stack trace

Describe the bug When the SDK throws an exception, the ToString() behavior is different from the behavior of other exceptions, in that it does not include the stack trace of the calling code. This means that when CosmosExceptions show up in our telemetry, we are never able to see what code outside of the SDK was calling the SDK

To Reproduce Given this code:

class Program
{
    static async Task Main(string[] args)
    {
        var client = new CosmosClientBuilder("xxx")
            .AddCustomHandlers(new MyHandler())
            .Build();

        var container = client.GetContainer("d", "c");
        await ExecuteQuery(container);
    }

    private static async Task ExecuteQuery(Container container)
    {
        var cts = new CancellationTokenSource(TimeSpan.FromSeconds(10));
        await container.GetItemQueryIterator<object>("select * from c").ReadNextAsync(cts.Token);
    }
}

internal class MyHandler : RequestHandler
{
    public override async Task<ResponseMessage> SendAsync(RequestMessage request, CancellationToken cancellationToken)
    {
        throw new InvalidOperationException("abc");
    }
}

If I run it, I get the following:

Unhandled exception. Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: InternalServerError (500); Substatus: 0; ActivityId: 00000000-0000-0000-0000-000000000000; Reason: (abc);
 ---> System.InvalidOperationException: abc
   at ConsoleApp5.MyHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken) in c:\path\ConsoleApp5\Program.cs:line 42
   at Microsoft.Azure.Cosmos.RequestHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(String resourceUriString, ResourceType resourceType, OperationType operationType, RequestOptions requestOptions, ContainerInternal cosmosContainerCore, Nullable`1 partitionKey, Stream streamPayload, Action`1 requestEnricher, CosmosDiagnosticsContext diagnosticsContext, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.CosmosQueryClientCore.ExecuteItemQueryAsync(String resourceUri, ResourceType resourceType, OperationType operationType, Guid clientQueryCorrelationId, QueryRequestOptions requestOptions, Action`1 queryPageDiagnostics, SqlQuerySpec sqlQuerySpec, String continuationToken, PartitionKeyRangeIdentity partitionKeyRange, Boolean isContinuationExpected, Int32 pageSize, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.CrossPartition.ExecuteQueryBasedOnFeedRangeVisitor.VisitAsync(FeedRangeEpk feedRange, Arguments argument, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Pagination.PartitionRangePageAsyncEnumerator`2.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Pagination.BufferedPartitionRangePageAsyncEnumerator`2.PrefetchAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Pagination.BufferedPartitionRangePageAsyncEnumerator`2.GetNextPageAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Pagination.PartitionRangePageAsyncEnumerator`2.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Pagination.CrossPartitionRangePageAsyncEnumerator`2.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.CrossPartition.Parallel.ParallelCrossPartitionQueryPipelineStage.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.LazyQueryPipelineStage.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.NameCacheStaleRetryQueryPipelineStage.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.CatchAllQueryPipelineStage.MoveNextAsync()
   --- End of inner exception stack trace ---
   at ConsoleApp5.MyHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken) in c:\path\ConsoleApp5\Program.cs:line 42
   at Microsoft.Azure.Cosmos.RequestHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(String resourceUriString, ResourceType resourceType, OperationType operationType, RequestOptions requestOptions, ContainerInternal cosmosContainerCore, Nullable`1 partitionKey, Stream streamPayload, Action`1 requestEnricher, CosmosDiagnosticsContext diagnosticsContext, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.CosmosQueryClientCore.ExecuteItemQueryAsync(String resourceUri, ResourceType resourceType, OperationType operationType, Guid clientQueryCorrelationId, QueryRequestOptions requestOptions, Action`1 queryPageDiagnostics, SqlQuerySpec sqlQuerySpec, String continuationToken, PartitionKeyRangeIdentity partitionKeyRange, Boolean isContinuationExpected, Int32 pageSize, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.CrossPartition.ExecuteQueryBasedOnFeedRangeVisitor.VisitAsync(FeedRangeEpk feedRange, Arguments argument, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Pagination.PartitionRangePageAsyncEnumerator`2.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Pagination.BufferedPartitionRangePageAsyncEnumerator`2.PrefetchAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Pagination.BufferedPartitionRangePageAsyncEnumerator`2.GetNextPageAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Pagination.PartitionRangePageAsyncEnumerator`2.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Pagination.CrossPartitionRangePageAsyncEnumerator`2.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.CrossPartition.Parallel.ParallelCrossPartitionQueryPipelineStage.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.LazyQueryPipelineStage.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.NameCacheStaleRetryQueryPipelineStage.MoveNextAsync()
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.CatchAllQueryPipelineStage.MoveNextAsync()
--- Cosmos Diagnostics ---{"DiagnosticVersion":"2","Summary":{"StartUtc":"2020-11-13T16:44:16.7583066Z","RunningElapsedTimeInMs":21.7877,"UserAgent":"cosmos-netstandard-sdk/3.14.0|3.14.0|03|X64|Microsoft Windows 10.0.19042|.NET Core 3.1.9|","TotalRequestCount":0,"FailedRequestCount":0},"Context":[]}

Note that Main and ExecuteQuery are not in the frames.

Expected behavior If instead my program looks like this:

class Program
{
    static async Task Main(string[] args)
    {
        await NonCosmosMethod();
    }

    private static async Task NonCosmosMethod()
    {
        throw new DivideByZeroException("abc");
    }
}

I get

Unhandled exception. System.DivideByZeroException: abc
   at ConsoleApp5.Program.NonCosmosMethod() in c:\path\ConsoleApp5\Program.cs:line 19
   at ConsoleApp5.Program.Main(String[] args) in c:\path\ConsoleApp5\Program.cs:line 14
   at ConsoleApp5.Program.<Main>(String[] args)

Actual behavior The stack trace is misleading (note that the trace is duplicated in the inner and outer exceptions. And the outer exception’s trace should extend outside of the SDK.

Environment summary SDK Version: 3.12 OS Version Version

[yufeih]

This issue causes a lot of pain in troubleshooting CosmosException. In case an CosmosException occurs, we (the SDK user) are mostly interested in how our code show up on the stack trace to pin point exactly where the exception occurs in our code. The stack trace returned today only shows Cosmos SDK code, which isn’t very useful to us. I also feel like carrying the original exception as InnerException is a better option than messing up with the stack trace.

[ccurrens]

Sure, the code that OP posted still seems to fail, but you can reproduce it without any custom handler by running an OrderBy query with no indexed on the ordered property. So, this code easily reproduces it:

class Program
{
    static async Task Main(string[] args)
    {
        try
        {
            const string Account = "https://localhost:8081";
            const string Key = "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==";
            using var client = new CosmosClient(Account, Key);
    
            var indexingPolicy = new IndexingPolicy();
            indexingPolicy.ExcludedPaths.Add(new ExcludedPath {Path = "/*"});
    
            await client.CreateDatabaseIfNotExistsAsync("TestDatabase");
            var database = client.GetDatabase("TestDatabase");
            await database.CreateContainerIfNotExistsAsync(new ContainerProperties
            {
                Id = "TestContainer",
                PartitionKeyPath = "/_pk",
                IndexingPolicy = indexingPolicy
            });
    
            var container = client.GetContainer("TestDatabase", "TestContainer");
            await container.GetItemQueryIterator<object>("select * from c ORDER BY c.LastChangedBy").ReadNextAsync();
        }
        catch (Exception e)
        {
            Console.WriteLine(e.ToString());
        }
    }
}

The outcome is an exception with the following ToString() message:

Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 554975ec-059f-4cc7-abf2-b6743096ca70; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 554975ec-059f-4cc7-abf2-b6743096ca70; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 554975ec-059f-4cc7-abf2-b6743096ca70; Reason: (Message: {"Errors":["The index path corresponding to the specified order-by item is excluded."]}
ActivityId: 554975ec-059f-4cc7-abf2-b6743096ca70, Request URI: /apps/DocDbApp/services/DocDbServer9/partitions/a4cb4955-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.19042 cosmos-netstandard-sdk/3.18.0);););
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.CrossPartition.OrderBy.OrderByCrossPartitionQueryPipelineStage.MoveNextAsync_Initialize_FromBeginningAsync(OrderByQueryPartitionRangePageAsyncEnumerator uninitializedEnumerator, ITrace trace)   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Pagination.PartitionRangePageAsyncEnumerator`2.MoveNextAsync(ITrace trace)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.CrossPartition.OrderBy.OrderByQueryPartitionRangePageAsyncEnumerator.GetNextPageAsync(ITrace trace, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Pagination.PartitionRangePageAsyncEnumerator`2.MoveNextAsync(ITrace trace)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Pagination.BufferedPartitionRangePageAsyncEnumerator`2.GetNextPageAsync(ITrace trace, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder.SetResult()
   at Microsoft.Azure.Cosmos.Pagination.BufferedPartitionRangePageAsyncEnumerator`2.PrefetchAsync(ITrace trace, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Pagination.PartitionRangePageAsyncEnumerator`2.MoveNextAsync(ITrace trace)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task.FinishStageTwo()
   at System.Threading.Tasks.Task.FinishSlow(Boolean userDelegateExecute)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread)
   at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

--- Cosmos Diagnostics --- (elided)

Note that there is no mention of the actual stack trace of the Main program. It seems to be due to the fact that CosmosException is often created with the constructor that takes in a stack trace string which came from a different exception, and this is overridden in CosmosException.StackTrace.

You can see the stack trace that should also have been printed if you add this in the catch block:

var st = new StackTrace(e, true);
// This prints the actual stack trace of the CosmosException from where it was thrown and caught by our code.
Console.WriteLine(st);

I was going to create a separate issue (and probably still will), but the way that CosmosExceptions are handled makes it difficult for us to have nice logging. Because stack traces are overridden in CosmosException, we can’t use something nice like Ben.Demystifier to give us more sane stack traces (it doesn’t work when the stack trace string is already materialized). Honestly, I feel like it would be much better if original exceptions were simply passed along with as an InnerException to CosmosException, so a) we can still demystify them and b) we get all stack traces and messages for the entire exception chain.