Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Response status code does not indicate success: ServiceUnavailable (503)
See original GitHub issueDescribe the bug
Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: ServiceUnavailable (503); Substatus: 0; ActivityId: f197457b-de2f-421a-951c-01b17311b1c3; Reason: (The request failed because the client was unable to establish connections to 4 endpoints across 1 regions. Please check for client resource starvation issues and verify connectivity between client and server.
ActivityId: f197457b-de2f-421a-951c-01b17311b1c3, Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, Windows/10.0.19043 cosmos-netstandard-sdk/3.19.3);
—> Microsoft.Azure.Documents.GoneException: The requested resource is no longer available at the server.
ActivityId: f197457b-de2f-421a-951c-01b17311b1c3, Windows/10.0.19043 cosmos-netstandard-sdk/3.19.3
at Microsoft.Azure.Documents.TimeoutHelper.ThrowGoneIfElapsed()
at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicasInternalAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicaAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
at Microsoft.Azure.Documents.ConsistencyReader.ReadSessionAsync(DocumentServiceRequest entity, ReadMode readMode)
at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync(Func1 callbackMethod, Func3 callShouldRetry, Func1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback) at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync(Func1 callbackMethod, Func3 callShouldRetry, Func1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback)
at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass30_0.<<InvokeAsync>b__0>d.MoveNext()
— End of stack trace from previous location —
at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable1 minBackoffForInBackoffCallback) --- End of inner exception stack trace --- at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable1 minBackoffForInBackoffCallback)
at Microsoft.Azure.Documents.StoreClient.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken, IRetryPolicy retryPolicy, Func`2 prepareRequestAsyncDelegate)
at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken);
Json Diagnosis
{
"name": "ReadItemAsync",
"id": "5885126c-8728-480c-97c4-f124443ef5ef",
"caller info": {
"member": "OperationHelperWithRootTraceAsync",
"file": "ClientContextCore.cs",
"line": 219
},
"start time": "02:12:10:056",
"duration in milliseconds": 30140.8303,
"data": {
"Client Configuration": {
"Client Created Time Utc": "2021-07-07T14:11:51.2798748Z",
"NumberOfClientsCreated": 1,
"User Agent": "cosmos-netstandard-sdk/3.20.0|3.19.3|0|X64|Microsoft Windows 10.0.19043|.NET 5.0.4|N|F 00000010|",
"ConnectionConfig": {
"gw": "(cps:50, urto:10, p:False, httpf: True)",
"rntbd": "(cto: 5, icto: 86400, mrpc: 30, mcpe: 65535, erd: False, pr: ReuseUnicastPort)",
"other": "(ed:False, be:False)"
},
"ConsistencyConfig": "(consistency: NotSet, prgns:[])"
}
},
"children": [
{
"name": "Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler",
"id": "e4abc4ae-e1bb-4034-9647-37e6e5eec8f7",
"start time": "02:12:10:063",
"duration in milliseconds": 30129.9626,
"children": [
{
"name": "Waiting for Initialization of client to complete",
"id": "03424e66-faac-43c2-bec4-5447076b3e0a",
"caller info": {
"member": "EnsureValidClientAsync",
"file": "DocumentClient.cs",
"line": 1416
},
"start time": "02:12:10:085",
"duration in milliseconds": 0.0834
},
{
"name": "Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler",
"id": "9d00add0-cec6-488e-b47f-20647615ac5c",
"start time": "02:12:10:088",
"duration in milliseconds": 30104.4814,
"data": {
"CPU Load History": {
"CPU History": "(2021-07-07T14:12:10.0956953Z 5.587)"
}
},
"children": [
{
"name": "Microsoft.Azure.Cosmos.Handlers.RetryHandler",
"id": "ae6b3434-9d7b-4343-9c43-2e9e9d93860c",
"start time": "02:12:10:096",
"duration in milliseconds": 30095.7494,
"children": [
{
"name": "Microsoft.Azure.Cosmos.Handlers.RouterHandler",
"id": "12680f7b-e21a-4018-9f2d-3bf79459b1a5",
"start time": "02:12:10:103",
"duration in milliseconds": 30075.9656,
"children": [
{
"name": "Microsoft.Azure.Cosmos.Handlers.TransportHandler",
"id": "c511edab-6320-4204-8d03-58e939faec92",
"start time": "02:12:10:104",
"duration in milliseconds": 30074.621,
"children": [
{
"name": "Microsoft.Azure.Documents.ServerStoreModel Transport Request",
"id": "b8ae7724-9f8b-4acc-a8d4-cf99963eb274",
"caller info": {
"member": "ProcessMessageAsync",
"file": "TransportHandler.cs",
"line": 109
},
"start time": "02:12:10:115",
"duration in milliseconds": 30058.0012,
"data": {
"Client Side Request Stats": {
"Id": "AggregatedClientSideRequestStatistics",
"ContactedReplicas": [
],
"RegionsContacted": [
"https://[DB Name]-eastus.documents.azure.com/"
],
"FailedReplicas": [
"rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14450/apps/f731575d-fb29-4640-9ffc-a6de234ddc91/services/0e39808c-b260-42d7-a738-4a6fdcedbd52/partitions/956def0c-664b-49e6-8fd9-93e741c6fbfe/replicas/132666141009659348s/",
"rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14392/apps/f731575d-fb29-4640-9ffc-a6de234ddc91/services/0e39808c-b260-42d7-a738-4a6fdcedbd52/partitions/956def0c-664b-49e6-8fd9-93e741c6fbfe/replicas/132666141064347521s/",
"rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14307/apps/f731575d-fb29-4640-9ffc-a6de234ddc91/services/0e39808c-b260-42d7-a738-4a6fdcedbd52/partitions/956def0c-664b-49e6-8fd9-93e741c6fbfe/replicas/132666141064347523p/",
"rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14398/apps/f731575d-fb29-4640-9ffc-a6de234ddc91/services/0e39808c-b260-42d7-a738-4a6fdcedbd52/partitions/956def0c-664b-49e6-8fd9-93e741c6fbfe/replicas/132666141064347522s/"
],
"AddressResolutionStatistics": [
{
"StartTimeUTC": "2021-07-07T14:12:10.3465783Z",
"EndTimeUTC": "2021-07-07T14:12:10.3537480Z",
"TargetEndpoint": "https://[DB Name]-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2f11RGAA%3d%3d%2fcolls%2f11RGAKPTz1w%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0"
},
{
"StartTimeUTC": "2021-07-07T14:12:34.1870639Z",
"EndTimeUTC": "2021-07-07T14:12:34.2479138Z",
"TargetEndpoint": "https://[DB Name]-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2f11RGAA%3d%3d%2fcolls%2f11RGAKPTz1w%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0"
}
],
"StoreResponseStatistics": [
{
"ResponseTimeUTC": "2021-07-07T14:12:16.2257465Z",
"ResourceType": "Document",
"OperationType": "Read",
"LocationEndpoint": "https://[DB Name]-eastus.documents.azure.com/",
"StoreResult": {
"ActivityId": "1e6193f5-056f-4332-83de-b6df560766e7",
"StatusCode": "Gone",
"SubStatusCode": "Unknown",
"LSN": -1,
"PartitionKeyRangeId": null,
"GlobalCommittedLSN": -1,
"ItemLSN": -1,
"UsingLocalLSN": true,
"QuorumAckedLSN": -1,
"SessionToken": null,
"CurrentWriteQuorum": -1,
"CurrentReplicaSetSize": -1,
"NumberOfReadRegions": -1,
"IsClientCpuOverloaded": false,
"IsValid": false,
"StorePhysicalAddress": "rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14450/apps/f731575d-fb29-4640-9ffc-a6de234ddc91/services/0e39808c-b260-42d7-a738-4a6fdcedbd52/partitions/956def0c-664b-49e6-8fd9-93e741c6fbfe/replicas/132666141009659348s/",
"RequestCharge": 0,
"BELatencyInMs": null,
"RntbdRequestStats": [
{
"EventName": "Created",
"StartTime": "2021-07-07T14:12:10.3787331Z",
"DurationInMicroSec": "5605"
},
{
"EventName": "ChannelAcquisitionStarted",
"StartTime": "2021-07-07T14:12:10.3843390Z",
"DurationInMicroSec": "5815116"
},
{
"EventName": "Failed",
"StartTime": "2021-07-07T14:12:16.1994550Z",
"DurationInMicroSec": "0"
}
],
"TransportException": "A client transport error occurred: The connection attempt timed out. (Time: 2021-07-07T14:12:16.1623911Z, activity ID: 1e6193f5-056f-4332-83de-b6df560766e7, error code: ConnectTimeout [0x0006], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14450/, connection: <not connected> -> rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14450/, payload sent: False, CPU history: (2021-07-07T14:12:01.2496391Z 0.730), (2021-07-07T14:12:11.2556371Z 15.034), CPU count: 8)"
}
},
{
"ResponseTimeUTC": "2021-07-07T14:12:22.1618391Z",
"ResourceType": "Document",
"OperationType": "Read",
"LocationEndpoint": "https://[DB Name]-eastus.documents.azure.com/",
"StoreResult": {
"ActivityId": "1e6193f5-056f-4332-83de-b6df560766e7",
"StatusCode": "Gone",
"SubStatusCode": "Unknown",
"LSN": -1,
"PartitionKeyRangeId": null,
"GlobalCommittedLSN": -1,
"ItemLSN": -1,
"UsingLocalLSN": true,
"QuorumAckedLSN": -1,
"SessionToken": null,
"CurrentWriteQuorum": -1,
"CurrentReplicaSetSize": -1,
"NumberOfReadRegions": -1,
"IsClientCpuOverloaded": false,
"IsValid": false,
"StorePhysicalAddress": "rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14392/apps/f731575d-fb29-4640-9ffc-a6de234ddc91/services/0e39808c-b260-42d7-a738-4a6fdcedbd52/partitions/956def0c-664b-49e6-8fd9-93e741c6fbfe/replicas/132666141064347521s/",
"RequestCharge": 0,
"BELatencyInMs": null,
"RntbdRequestStats": [
{
"EventName": "Created",
"StartTime": "2021-07-07T14:12:16.2289170Z",
"DurationInMicroSec": "27"
},
{
"EventName": "ChannelAcquisitionStarted",
"StartTime": "2021-07-07T14:12:16.2289443Z",
"DurationInMicroSec": "5922986"
},
{
"EventName": "Failed",
"StartTime": "2021-07-07T14:12:22.1519305Z",
"DurationInMicroSec": "0"
}
],
"TransportException": "A client transport error occurred: The connection attempt timed out. (Time: 2021-07-07T14:12:22.1453111Z, activity ID: 1e6193f5-056f-4332-83de-b6df560766e7, error code: ConnectTimeout [0x0006], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14392/, connection: <not connected> -> rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14392/, payload sent: False, CPU history: (2021-07-07T14:12:01.2496391Z 0.730), (2021-07-07T14:12:11.2556371Z 15.034), (2021-07-07T14:12:21.2544907Z 17.808), CPU count: 8)"
}
},
{
"ResponseTimeUTC": "2021-07-07T14:12:28.1709010Z",
"ResourceType": "Document",
"OperationType": "Read",
"LocationEndpoint": "https://[DB Name]-eastus.documents.azure.com/",
"StoreResult": {
"ActivityId": "1e6193f5-056f-4332-83de-b6df560766e7",
"StatusCode": "Gone",
"SubStatusCode": "Unknown",
"LSN": -1,
"PartitionKeyRangeId": null,
"GlobalCommittedLSN": -1,
"ItemLSN": -1,
"UsingLocalLSN": true,
"QuorumAckedLSN": -1,
"SessionToken": null,
"CurrentWriteQuorum": -1,
"CurrentReplicaSetSize": -1,
"NumberOfReadRegions": -1,
"IsClientCpuOverloaded": false,
"IsValid": false,
"StorePhysicalAddress": "rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14307/apps/f731575d-fb29-4640-9ffc-a6de234ddc91/services/0e39808c-b260-42d7-a738-4a6fdcedbd52/partitions/956def0c-664b-49e6-8fd9-93e741c6fbfe/replicas/132666141064347523p/",
"RequestCharge": 0,
"BELatencyInMs": null,
"RntbdRequestStats": [
{
"EventName": "Created",
"StartTime": "2021-07-07T14:12:22.1620213Z",
"DurationInMicroSec": "15"
},
{
"EventName": "ChannelAcquisitionStarted",
"StartTime": "2021-07-07T14:12:22.1620367Z",
"DurationInMicroSec": "5998179"
},
{
"EventName": "Failed",
"StartTime": "2021-07-07T14:12:28.1602163Z",
"DurationInMicroSec": "0"
}
],
"TransportException": "A client transport error occurred: The connection attempt timed out. (Time: 2021-07-07T14:12:28.1524973Z, activity ID: 1e6193f5-056f-4332-83de-b6df560766e7, error code: ConnectTimeout [0x0006], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14307/, connection: <not connected> -> rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14307/, payload sent: False, CPU history: (2021-07-07T14:12:01.2496391Z 0.730), (2021-07-07T14:12:11.2556371Z 15.034), (2021-07-07T14:12:21.2544907Z 17.808), CPU count: 8)"
}
},
{
"ResponseTimeUTC": "2021-07-07T14:12:34.1637008Z",
"ResourceType": "Document",
"OperationType": "Read",
"LocationEndpoint": "https://[DB Name]-eastus.documents.azure.com/",
"StoreResult": {
"ActivityId": "1e6193f5-056f-4332-83de-b6df560766e7",
"StatusCode": "Gone",
"SubStatusCode": "Unknown",
"LSN": -1,
"PartitionKeyRangeId": null,
"GlobalCommittedLSN": -1,
"ItemLSN": -1,
"UsingLocalLSN": true,
"QuorumAckedLSN": -1,
"SessionToken": null,
"CurrentWriteQuorum": -1,
"CurrentReplicaSetSize": -1,
"NumberOfReadRegions": -1,
"IsClientCpuOverloaded": false,
"IsValid": false,
"StorePhysicalAddress": "rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14398/apps/f731575d-fb29-4640-9ffc-a6de234ddc91/services/0e39808c-b260-42d7-a738-4a6fdcedbd52/partitions/956def0c-664b-49e6-8fd9-93e741c6fbfe/replicas/132666141064347522s/",
"RequestCharge": 0,
"BELatencyInMs": null,
"RntbdRequestStats": [
{
"EventName": "Created",
"StartTime": "2021-07-07T14:12:28.1709211Z",
"DurationInMicroSec": "16"
},
{
"EventName": "ChannelAcquisitionStarted",
"StartTime": "2021-07-07T14:12:28.1709371Z",
"DurationInMicroSec": "5982380"
},
{
"EventName": "Failed",
"StartTime": "2021-07-07T14:12:34.1533173Z",
"DurationInMicroSec": "0"
}
],
"TransportException": "A client transport error occurred: The connection attempt timed out. (Time: 2021-07-07T14:12:34.1458080Z, activity ID: 1e6193f5-056f-4332-83de-b6df560766e7, error code: ConnectTimeout [0x0006], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14398/, connection: <not connected> -> rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14398/, payload sent: False, CPU history: (2021-07-07T14:12:01.2496391Z 0.730), (2021-07-07T14:12:11.2556371Z 15.034), (2021-07-07T14:12:21.2544907Z 17.808), (2021-07-07T14:12:31.2612624Z 2.534), CPU count: 8)"
}
},
{
"ResponseTimeUTC": "2021-07-07T14:12:40.1591328Z",
"ResourceType": "Document",
"OperationType": "Read",
"LocationEndpoint": "https://[DB Name]-eastus.documents.azure.com/",
"StoreResult": {
"ActivityId": "1e6193f5-056f-4332-83de-b6df560766e7",
"StatusCode": "Gone",
"SubStatusCode": "Unknown",
"LSN": -1,
"PartitionKeyRangeId": null,
"GlobalCommittedLSN": -1,
"ItemLSN": -1,
"UsingLocalLSN": true,
"QuorumAckedLSN": -1,
"SessionToken": null,
"CurrentWriteQuorum": -1,
"CurrentReplicaSetSize": -1,
"NumberOfReadRegions": -1,
"IsClientCpuOverloaded": false,
"IsValid": false,
"StorePhysicalAddress": "rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14398/apps/f731575d-fb29-4640-9ffc-a6de234ddc91/services/0e39808c-b260-42d7-a738-4a6fdcedbd52/partitions/956def0c-664b-49e6-8fd9-93e741c6fbfe/replicas/132666141064347522s/",
"RequestCharge": 0,
"BELatencyInMs": null,
"RntbdRequestStats": [
{
"EventName": "Created",
"StartTime": "2021-07-07T14:12:34.2502567Z",
"DurationInMicroSec": "159"
},
{
"EventName": "ChannelAcquisitionStarted",
"StartTime": "2021-07-07T14:12:34.2504164Z",
"DurationInMicroSec": "5898971"
},
{
"EventName": "Failed",
"StartTime": "2021-07-07T14:12:40.1493879Z",
"DurationInMicroSec": "0"
}
],
"TransportException": "A client transport error occurred: The connection attempt timed out. (Time: 2021-07-07T14:12:40.1402251Z, activity ID: 1e6193f5-056f-4332-83de-b6df560766e7, error code: ConnectTimeout [0x0006], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14398/, connection: <not connected> -> rntbd://cdb-ms-prod-eastus1-fd58.documents.azure.com:14398/, payload sent: False, CPU history: (2021-07-07T14:12:01.2496391Z 0.730), (2021-07-07T14:12:11.2556371Z 15.034), (2021-07-07T14:12:21.2544907Z 17.808), (2021-07-07T14:12:31.2612624Z 2.534), CPU count: 8)"
}
}
],
"HttpResponseStats": [
{
"StartTimeUTC": "2021-07-07T14:12:10.2438278Z",
"DurationInMs": 0.0004,
"RequestUri": "https://[DB Name]-eastus.documents.azure.com/dbs/useraccounts/colls/[ContainerName]",
"ResourceType": "Collection",
"HttpMethod": "GET",
"ActivityId": "3aeede29-1b23-4d6d-b23b-b41f98356003",
"StatusCode": "OK"
},
{
"StartTimeUTC": "2021-07-07T14:12:10.2964144Z",
"DurationInMs": 0.0004,
"RequestUri": "https://[DB Name]-eastus.documents.azure.com/dbs/11RGAA==/colls/11RGAKPTz1w=/pkranges",
"ResourceType": "PartitionKeyRange",
"HttpMethod": "GET",
"ActivityId": "4b0551e8-d42d-44d8-a437-025a942f9a6b",
"StatusCode": "OK"
},
{
"StartTimeUTC": "2021-07-07T14:12:10.3237729Z",
"DurationInMs": 0.0004,
"RequestUri": "https://[DB Name]-eastus.documents.azure.com/dbs/11RGAA==/colls/11RGAKPTz1w=/pkranges",
"ResourceType": "PartitionKeyRange",
"HttpMethod": "GET",
"ActivityId": "8a28fd4f-4275-4c37-8b3b-0a37649dcdbd",
"StatusCode": "NotModified",
"ReasonPhrase": "Not Modified"
},
{
"StartTimeUTC": "2021-07-07T14:12:10.3527602Z",
"DurationInMs": 0.0005,
"RequestUri": "https://[DB Name]-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2f11RGAA%3d%3d%2fcolls%2f11RGAKPTz1w%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0",
"ResourceType": "Document",
"HttpMethod": "GET",
"ActivityId": "1e6193f5-056f-4332-83de-b6df560766e7",
"StatusCode": "OK"
},
{
"StartTimeUTC": "2021-07-07T14:12:34.2478534Z",
"DurationInMs": 0.0004,
"RequestUri": "https://[DB Name]-eastus.documents.azure.com//addresses/?$resolveFor=dbs%2f11RGAA%3d%3d%2fcolls%2f11RGAKPTz1w%3d%2fdocs&$filter=protocol eq rntbd&$partitionKeyRangeIds=0",
"ResourceType": "Document",
"HttpMethod": "GET",
"ActivityId": "1e6193f5-056f-4332-83de-b6df560766e7",
"StatusCode": "OK"
}
]
},
"Point Operation Statistics": {
"Id": "PointOperationStatistics",
"ActivityId": "1e6193f5-056f-4332-83de-b6df560766e7",
"ResponseTimeUtc": "2021-07-07T14:12:40.1773023Z",
"StatusCode": 503,
"SubStatusCode": 0,
"RequestCharge": 0,
"RequestUri": "dbs/useraccounts/colls/[ContainerName]/docs/d091f8de-473a-497e-b42b-46dd3f505923",
"ErrorMessage": "Microsoft.Azure.Documents.ServiceUnavailableException: The request failed because the client was unable to establish connections to 4 endpoints across 1 regions. Please check for client resource starvation issues and verify connectivity between client and server.\r\nActivityId: 1e6193f5-056f-4332-83de-b6df560766e7, Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, Windows/10.0.19043 cosmos-netstandard-sdk/3.19.3\r\n ---> Microsoft.Azure.Documents.GoneException: The requested resource is no longer available at the server.\r\nActivityId: 1e6193f5-056f-4332-83de-b6df560766e7, Windows/10.0.19043 cosmos-netstandard-sdk/3.19.3\r\n at Microsoft.Azure.Documents.TimeoutHelper.ThrowGoneIfElapsed()\r\n at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicasInternalAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)\r\n at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicaAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)\r\n at Microsoft.Azure.Documents.ConsistencyReader.ReadSessionAsync(DocumentServiceRequest entity, ReadMode readMode)\r\n at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync(Func`1 callbackMethod, Func`3 callShouldRetry, Func`1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)\r\n at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync(Func`1 callbackMethod, Func`3 callShouldRetry, Func`1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)\r\n at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass30_0.<<InvokeAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func`1 executeAsync, Func`1 prepareRequest, IRequestRetryPolicy`2 policy, CancellationToken cancellationToken, Func`1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback)\r\n --- End of inner exception stack trace ---\r\n at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func`1 executeAsync, Func`1 prepareRequest, IRequestRetryPolicy`2 policy, CancellationToken cancellationToken, Func`1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback)\r\n at Microsoft.Azure.Documents.StoreClient.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken, IRetryPolicy retryPolicy, Func`2 prepareRequestAsyncDelegate)\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)",
"RequestSessionToken": null,
"ResponseSessionToken": null,
"BELatencyInMs": null
}
}
}
]
}
]
}
]
}
]
}
]
}
]
}
To Reproduce
var cosmos = new CosmosClient("endpointUrl", "authorizationKey");
var container = cosmos .GetContainer("databaseName", "containerName");
var result = await container.ReadItemAsync<UserModel>("123", new PartitionKey("123"));
Environment summary SDK Version: Assembly Microsoft.Azure.Cosmos.Client, Version=3.20.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35 OS Version : Windows-10
Additional context
I am unable to connect CosmosDb using Direct connection mode. After debugging I observed that the API is hitting different ports and URLs under the hood, which are not enabled in our company firewall level.
The temporary work around we did was enabling cdb-ms-prod-eastus1-fd58.documents.azure.com with random ports. It worked only after opening LAN to WAN to cdb-ms-prod-eastus1-fd58.documents.azure.com on the firewall.
Please recommend how to handle this issue.
What ports need to be open on a firewall to access cosmos API?
Thank you Deepu
Issue Analytics
- State:
- Created 2 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
Microsoft.Azure.Cosmos.Client: Response status code ...
Microsoft.Azure.Cosmos.Client: Response status code does not indicate success: ServiceUnavailable (503) ... I am trying to add new items into the ...
Read more >Troubleshoot Azure Cosmos DB service unavailable ...
In certain conditions, the HTTP 503 Service Unavailable error includes a substatus code that helps to identify the cause.
Read more >503 Service Unavailable - HTTP - MDN Web Docs
The HyperText Transfer Protocol (HTTP) 503 Service Unavailable server error response code indicates that the server is not ready to handle ...
Read more >503 Service Unavailable Error: What It Is and How to Fix It
A 503 Service Unavailable Error is an HTTP response status code indicating that a server is temporarily unable to handle the request.
Read more >How to Fix the HTTP Error 503 Service Unavailable (6 Steps)
The HTTP Error 503 status code indicates that the server is currently unable to handle the request. Find out more about the causes...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
The troubleshooting links are added to the error messages. Closing this issue since the initial question has been answered.
Yes, private VNETs solution may works. Will try and keep you posted.
Thanks @j82w