403: Forbidden Response when AllowBulkExecution = true + Resource Token

Describe the bug When retrieving a resource token to make api calls to Azure Cosmos DB, and the client is created with the AllowBulkExecution = true, then Azure Cosmos DB returns a 403: Forbidden error on an api call. The same options work fine when using the local CosmosDB emulator. The call api calls also work fine if AllowBulkExecution setting is false. I’ve checked my azure CosmosDB firewall settings: it’s set to allow all networks.

To Reproduce Can be reproduced using this project: https://github.com/InquisitorJax/Xamarin-Cosmos-DB Check the readme file on setting up the local.settings.json file to retrieve the resource token from an azure instance.

Expected behavior Api calls should be successful.

Actual behavior 403: Forbidden Error is returned:

Environment summary SDK Version: 3.12.0 OS Version (e.g. Windows, Linux, MacOSX) Windows

Additional context Stack Trace: {Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 0; ActivityId: ; Reason: (); at Microsoft.Azure.Cosmos.ResponseMessage.EnsureSuccessStatusCode () [0x0000e] in <250931dbc7a64174b3f1ed93d3081ffb>:0 at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.ProcessMessage[T] (Microsoft.Azure.Cosmos.ResponseMessage responseMessage, System.Func2[T,TResult] createResponse) [0x00002] in <250931dbc7a64174b3f1ed93d3081ffb>:0 at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.CreateItemResponse[T] (Microsoft.Azure.Cosmos.ResponseMessage responseMessage) [0x00000] in <250931dbc7a64174b3f1ed93d3081ffb>:0 at Microsoft.Azure.Cosmos.ContainerCore.CreateItemAsync[T] (Microsoft.Azure.Cosmos.CosmosDiagnosticsContext diagnosticsContext, T item, System.Nullable1[T] partitionKey, Microsoft.Azure.Cosmos.ItemRequestOptions requestOptions, System.Threading.CancellationToken cancellationToken) [0x000b0] in <250931dbc7a64174b3f1ed93d3081ffb>:0 at Microsoft.Azure.Cosmos.ClientContextCore.RunWithDiagnosticsHelperAsync[TResult] (Microsoft.Azure.Cosmos.CosmosDiagnosticsContext diagnosticsContext, System.Func2[T,TResult] task) [0x0009e] in <250931dbc7a64174b3f1ed93d3081ffb>:0 at XamarinCosmosDB.CosmosRepository.SaveModelAsync[T] (T model) [0x0009e] in D:\DEV\Git\Xamarin-Cosmos-DB\XamarinCosmosDB\XamarinCosmosDB\CosmosRepository.cs:137 --- Cosmos Diagnostics ---{"DiagnosticVersion":"2","Summary":{"StartUtc":"2020-08-17T21:32:30.8715930Z","TotalElapsedTimeInMs":12290.9743,"UserAgent":"cosmos-netstandard-sdk/3.12.0|3.11.4|02|X86|Unix 4.14.112.0|Mono 6.12.0 2020-02 83105ba2246 |F 00000001|","TotalRequestCount":2,"FailedRequestCount":2},"Context":[{"Id":"SynchronizationContext","ElapsedTimeInMs":11.1346},{"Id":"ItemSerialize","ElapsedTimeInMs":134.708},{"Id":"UsingWaitAsync","ElapsedTimeInMs":0.8897},{"Id":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","HandlerElapsedTimeInMs":3558.1352},{"Id":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","HandlerElapsedTimeInMs":3504.7875000000004},{"Id":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","HandlerElapsedTimeInMs":3464.9487000000004},{"Id":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","HandlerElapsedTimeInMs":127.7205},{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-fd42.documents.azure.com:14104/apps/814e7582-0a8b-498b-8d4f-92263e029030/services/f2d1101a-4d64-4d76-bfa7-e60d2dcfe6c4/partitions/22211ae2-1228-4497-bdbd-d913f933812e/replicas/132368769092028206s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-fd42.documents.azure.com:14432/apps/814e7582-0a8b-498b-8d4f-92263e029030/services/f2d1101a-4d64-4d76-bfa7-e60d2dcfe6c4/partitions/22211ae2-1228-4497-bdbd-d913f933812e/replicas/132389420512205363s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-eastus1-fd42.documents.azure.com:14148/apps/814e7582-0a8b-498b-8d4f-92263e029030/services/f2d1101a-4d64-4d76-bfa7-e60d2dcfe6c4/partitions/22211ae2-1228-4497-bdbd-d913f933812e/replicas/132415566475158421s/"}],"RegionsContacted":["https://remotime-eastus.documents.azure.com:443/"],"FailedReplicas":[]},{"Id":"Microsoft.Azure.Documents.ServerStoreModel","ElapsedTimeInMs":3401.2183},{"Id":"AddressResolutionStatistics","StartTimeUtc":"2020-08-17T21:32:39.6743060Z","EndTimeUtc":"2020-08-17T21:32:39.9263650Z","ElapsedTimeInMs":252.05900000000003,"TargetEndpoint":"https://remotime-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2fmhwpAA%3d%3d%2fcolls%2fmhwpAJ28n9Q%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0"},{"Id":"StoreResponseStatistics","StartTimeUtc":"2020-08-17T21:32:39.6010870Z","ResponseTimeUtc":"2020-08-17T21:32:42.8359020Z","ElapsedTimeInMs":3234.815,"ResourceType":"Document","OperationType":"Batch","LocationEndpoint":"https://remotime-eastus.documents.azure.com:443/","ActivityId":"db273de7-94b4-48a2-9ebc-1a519c5f13fe","StoreResult":"StorePhysicalAddress: rntbd://cdb-ms-prod-eastus1-fd42.documents.azure.com:14063/apps/814e7582-0a8b-498b-8d4f-92263e029030/services/f2d1101a-4d64-4d76-bfa7-e60d2dcfe6c4/partitions/22211ae2-1228-4497-bdbd-d913f933812e/replicas/132415566475158422p/, LSN: 57, GlobalCommittedLsn: 57, PartitionKeyRangeId: , IsValid: True, StatusCode: 403, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: -1#57, UsingLocalLSN: False, TransportException: null"},{"Id":"PointOperationStatistics","ActivityId":"db273de7-94b4-48a2-9ebc-1a519c5f13fe","ResponseTimeUtc":"2020-08-17T21:32:43.0081550Z","StatusCode":403,"SubStatusCode":0,"RequestCharge":0.0,"RequestUri":"dbs/remotimedb/colls/UserData","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Request is blocked. Please check your authorization token and Cosmos DB account firewall settings.\"]}\nActivityId: db273de7-94b4-48a2-9ebc-1a519c5f13fe, Request URI: /apps/814e7582-0a8b-498b-8d4f-92263e029030/services/f2d1101a-4d64-4d76-bfa7-e60d2dcfe6c4/partitions/22211ae2-1228-4497-bdbd-d913f933812e/replicas/132415566475158422p/, RequestStats: Please see CosmosDiagnostics, SDK: Linux/Unknown cosmos-netstandard-sdk/3.11.4\n at Microsoft.Azure.Documents.TransportClient.ThrowServerException (System.String resourceAddress, Microsoft.Azure.Documents.StoreResponse storeResponse, System.Uri physicalAddress, System.Guid activityId, Microsoft.Azure.Documents.DocumentServiceRequest request) [0x004d2] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.Rntbd.TransportClient.InvokeStoreAsync (System.Uri physicalAddress, Microsoft.Azure.Documents.ResourceOperation resourceOperation, Microsoft.Azure.Documents.DocumentServiceRequest request) [0x003c8] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.StoreResult.ToResponse (Microsoft.Azure.Documents.RequestChargeTracker requestChargeTracker) [0x0004f] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.ConsistencyWriter.WritePrivateAsync (Microsoft.Azure.Documents.DocumentServiceRequest request, Microsoft.Azure.Documents.TimeoutHelper timeout, System.Boolean forceRefresh) [0x00573] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.BackoffRetryUtility1[T].ExecuteRetryAsync (System.Func1[TResult] callbackMethod, System.Func3[T1,T2,TResult] callShouldRetry, System.Func1[TResult] inBackoffAlternateCallbackMethod, System.TimeSpan minBackoffForInBackoffCallback, System.Threading.CancellationToken cancellationToken, System.Action1[T] preRetryCallback) [0x00096] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying (System.Runtime.ExceptionServices.ExceptionDispatchInfo capturedException) [0x00011] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.BackoffRetryUtility1[T].ExecuteRetryAsync (System.Func1[TResult] callbackMethod, System.Func3[T1,T2,TResult] callShouldRetry, System.Func1[TResult] inBackoffAlternateCallbackMethod, System.TimeSpan minBackoffForInBackoffCallback, System.Threading.CancellationToken cancellationToken, System.Action1[T] preRetryCallback) [0x001bf] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.ConsistencyWriter.WriteAsync (Microsoft.Azure.Documents.DocumentServiceRequest entity, Microsoft.Azure.Documents.TimeoutHelper timeout, System.Boolean forceRefresh, System.Threading.CancellationToken cancellationToken) [0x00102] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.ReplicatedResourceClient+<>c__DisplayClass26_0.<InvokeAsync>b__0 (Microsoft.Azure.Documents.GoneAndRetryRequestRetryPolicyContext contextArguments) [0x00187] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse] (System.Func1[TResult] executeAsync, System.Func1[TResult] prepareRequest, Microsoft.Azure.Documents.IRequestRetryPolicy2[TRequest,TResponse] policy, System.Threading.CancellationToken cancellationToken, System.Func1[TResult] inBackoffAlternateCallbackMethod, System.Nullable1[T] minBackoffForInBackoffCallback) [0x000df] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying (System.Runtime.ExceptionServices.ExceptionDispatchInfo capturedException) [0x00011] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse] (System.Func1[TResult] executeAsync, System.Func1[TResult] prepareRequest, Microsoft.Azure.Documents.IRequestRetryPolicy2[TRequest,TResponse] policy, System.Threading.CancellationToken cancellationToken, System.Func1[TResult] inBackoffAlternateCallbackMethod, System.Nullable1[T] minBackoffForInBackoffCallback) [0x0028b] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.StoreClient.ProcessMessageAsync (Microsoft.Azure.Documents.DocumentServiceRequest request, System.Threading.CancellationToken cancellationToken, Microsoft.Azure.Documents.IRetryPolicy retryPolicy, System.Func2[T,TResult] prepareRequestAsyncDelegate) [0x002fd] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Documents.ServerStoreModel.ProcessMessageAsync (Microsoft.Azure.Documents.DocumentServiceRequest request, System.Threading.CancellationToken cancellationToken) [0x00165] in <18285060bc1144e3a326a50b67232825>:0 \n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync (Microsoft.Azure.Cosmos.RequestMessage request, System.Threading.CancellationToken cancellationToken) [0x0019d] in <250931dbc7a64174b3f1ed93d3081ffb>:0 \n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync (Microsoft.Azure.Cosmos.RequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00074] in <250931dbc7a64174b3f1ed93d3081ffb>:0 ",“RequestSessionToken”:null,“ResponseSessionToken”:“0:-1#57”},{“Id”:“BatchAsyncContainerExecutor.ToResponse”,“ElapsedTimeInMs”:13.0839}]}}

[InquisitorJax]

Thanks for the feedback @ealsur - would it then make sense to have an error response that is a little more informative than “forbidden”? I can imagine people making PK scoped permissions, and thinking turning on bulk operations flag on a request will improve performance… would be nice if response error / documentation was clear that this is not a supported scenario by design.

[ealsur]

Your point is correct, what I’m saying is that the protocol used in this case does not send the PartitionKey header, because technically the Bulk requests could include documents from different Partition Keys that are stored in the same physical partition. Since we don’t send the PartitionKey header, even though all documents are for the same PartitionKey, the validation process that happens on the backend side is rejecting the request with a 403, because the Permission is set for a particular PartitionKey (and the request does not have the header).