Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

RequestCharge regression in SDK 3.7.0

See original GitHub issue

Describe the bug RequestCharge regression when updating the SDK from v3.6.0 => v3.7.0

The complex query that has the issue :

SELECT DISTINCT VALUE {{
    CompanyName : company.Name,
    Countries : ARRAY(
        SELECT  VALUE address.Country
        FROM    address IN company.OfficeAddresses
    )
}}
FROM    c.Document company
OFFSET 0 LIMIT 20

The issue is also happening if I remove OFFSET 0 LIMIT 20 and If I add SELECT DISTINCT TOP 20 VALUE

To Reproduce I have pushed a repro on my Github : https://github.com/YohanSciubukgian/CosmosDbConnector/blob/master/src/TestCosmosDbFacade/TestCosmosDbConnector.cs#L77

Expected behavior SDK 3.7.0 should have the same RequestCharge in comparison wit the SDK 3.6.0 for the exacte same query & using the exacte same CosmosDB container. This issue has been seen using CosmosDB on Azure & the behavior also happens using the CosmosDB Emulator.

Actual behavior SDK 3.7.0 RequestCharge = 173.36 RU SDK 3.6.0 RequestCharge = 9.34 RU

Environment summary SDK Version: 3.7.0 OS Version : Windows 10 (1909 18363.752) CosmosDB : 10’000 RU on Azure CosmosDB Emulator Version : 2.9.2

Additional context

SDK v3.6.0 RequestCharge => 9.34 Full Diagnostics :

{"Summary":{"StartUtc":"2020-04-03T17:04:56.8999242Z","ElapsedTime":"00:00:00.0066586","UserAgent":"cosmos-netstandard-sdk/3.6.0|3.4.2|83740|X64|Microsoft Windows 10.0.18363|.NET Core 3.1.3|"},"Context":[{"Id":"QueryReadNextAsync","ElapsedTime":"00:00:00.1245243"},{"PKRangeId":"0","QueryMetric":"totalExecutionTimeInMs=1.10;queryCompileTimeInMs=0.17;queryLogicalPlanBuildTimeInMs=0.04;queryPhysicalPlanBuildTimeInMs=0.05;queryOptimizationTimeInMs=0.00;VMExecutionTimeInMs=0.66;indexLookupTimeInMs=0.21;documentLoadTimeInMs=0.20;systemFunctionExecuteTimeInMs=0.00;userFunctionExecuteTimeInMs=0.00;retrievedDocumentCount=20;retrievedDocumentSize=24220;outputDocumentCount=20;outputDocumentSize=5299;writeOutputTimeInMs=0.02;indexUtilizationRatio=1.00","IndexUtilization":"","SchedulingTimeSpan":{"TurnaroundTimeInMs":"22.3636","ResponseTimeInMs":"14.3668","RunTimeInMs":"7.3617","WaitTimeInMs":"15.0019","NumberOfPreemptions":"1"},"Context":[{"Id":"RequestInvokerHandler","ElapsedTime":"00:00:00.0066586"},{"Id":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","ElapsedTime":"00:00:00.0053495"},{"Id":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","ElapsedTime":"00:00:00.0053371"},{"Id":"TransportHandler","ElapsedTime":"00:00:00.0053344"},{"Id":"PointOperationStatistics","ActivityId":"bbeaa60b-2a55-4968-81eb-38611edb8693","StatusCode":200,"SubStatusCode":0,"RequestCharge":9.34,"RequestUri":"dbs/test_database_v3/colls/test_collection_v3","RequestSessionToken":null,"ResponseSessionToken":"0:-1#111965","ClientRequestStats":{"RequestStartTimeUtc":"2020-04-03T17:04:56.9917317Z","RequestEndTimeUtc":"2020-04-03T17:04:56.9965174Z","RequestLatency":"00:00:00.0047857","IsCpuOverloaded":false,"NumberRegionsAttempted":1,"ResponseStatisticsList":[{"ResponseTime":"2020-04-03T17:04:56.9965043Z","ResourceType":2,"OperationType":15,"StoreResult":"StorePhysicalAddress: rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: 111965, GlobalCommittedLsn: -1, PartitionKeyRangeId: 0, IsValid: True, StatusCode: 200, SubStatusCode: 0, RequestCharge: 4.67, ItemLSN: -1, SessionToken: -1#111965, UsingLocalLSN: False, TransportException: null"},{"ResponseTime":"2020-04-03T17:04:56.9965174Z","ResourceType":2,"OperationType":15,"StoreResult":"StorePhysicalAddress: rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: 111965, GlobalCommittedLsn: -1, PartitionKeyRangeId: 0, IsValid: True, StatusCode: 200, SubStatusCode: 0, RequestCharge: 4.67, ItemLSN: -1, SessionToken: -1#111965, UsingLocalLSN: False, TransportException: null"}],"AddressResolutionStatistics":[],"SupplementalResponseStatistics":[],"FailedReplicas":[],"RegionsContacted":["https://127.0.0.1:8081/"],"ContactedReplicas":["rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/"]}}]}]}

SDK 3.7.0 RequestCharge : 173.36 Full Diagnostics :

{"Summary":{"StartUtc":"2020-04-03T17:05:44.7496452Z","TotalElapsedTime":"00:00:00.2389366","UserAgent":"cosmos-netstandard-sdk/3.7.0|3.7.1|04818|X64|Microsoft Windows 10.0.18363|.NET Core 3.1.3|","TotalRequestCount":2,"FailedRequestCount":0},"Context":[{"Id":"CreateQueryPipeline","ElapsedTime":"00:00:00.1332024"},{"Id":"ServiceInterop","ElapsedTime":"00:00:00.0877450"},{"PKRangeId":"0","StartUtc":"2020-04-03T17:05:44.8993377Z","QueryMetric":"totalExecutionTimeInMs=17.73;queryCompileTimeInMs=0.12;queryLogicalPlanBuildTimeInMs=0.03;queryPhysicalPlanBuildTimeInMs=0.05;queryOptimizationTimeInMs=0.00;VMExecutionTimeInMs=17.28;indexLookupTimeInMs=0.23;documentLoadTimeInMs=7.20;systemFunctionExecuteTimeInMs=0.00;userFunctionExecuteTimeInMs=0.00;retrievedDocumentCount=1000;retrievedDocumentSize=1255480;outputDocumentCount=1000;outputDocumentSize=277841;writeOutputTimeInMs=0.70;indexUtilizationRatio=1.00","IndexUtilization":"","Context":[{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":2,"Uri":"rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/"}],"RegionsContacted":["https://127.0.0.1:8081/"],"FailedReplicas":[]},{"Id":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","ElapsedTime":"00:00:00.0402410"},{"Id":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","ElapsedTime":"00:00:00.0402322"},{"Id":"TransportHandler","ElapsedTime":"00:00:00.0402290"},{"Id":"StoreResponseStatistics","ResponseTimeUtc":"2020-04-03T17:05:44.9407789Z","ResourceType":"Document","OperationType":"Query","LocationEndpoint":"https://127.0.0.1:8081/","StoreResult":"StorePhysicalAddress: rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: 113981, GlobalCommittedLsn: -1, PartitionKeyRangeId: 0, IsValid: True, StatusCode: 200, SubStatusCode: 0, RequestCharge: 86.68, ItemLSN: -1, SessionToken: -1#113981, UsingLocalLSN: False, TransportException: null"},{"Id":"StoreResponseStatistics","ResponseTimeUtc":"2020-04-03T17:05:44.9407876Z","ResourceType":"Document","OperationType":"Query","LocationEndpoint":"https://127.0.0.1:8081/","StoreResult":"StorePhysicalAddress: rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: 113981, GlobalCommittedLsn: -1, PartitionKeyRangeId: 0, IsValid: True, StatusCode: 200, SubStatusCode: 0, RequestCharge: 86.68, ItemLSN: -1, SessionToken: -1#113981, UsingLocalLSN: False, TransportException: null"}]}]}

Query Metrics comparison :

SDK Version	3.6.0	3.7.0
totalExecutionTimeInMs	1.1	17.73
queryCompileTimeInMs	0.17	0.12
queryLogicalPlanBuildTimeInMs	0.04	0.03
queryPhysicalPlanBuildTimeInMs	0.05	0.05
queryOptimizationTimeInMs	0	0
VMExecutionTimeInMs	0.66	17.28
indexLookupTimeInMs	0.21	0.23
documentLoadTimeInMs	0.2	7.2
systemFunctionExecuteTimeInMs	0	0
userFunctionExecuteTimeInMs	0	0
retrievedDocumentCount	20	1000
retrievedDocumentSize	24220	1255480
outputDocumentCount	20	1000
outputDocumentSize	5299	277841
writeOutputTimeInMs	0.02	0.7
indexUtilizationRatio	1	1

Issue Analytics

State:
Created 3 years ago
Comments:5 (2 by maintainers)

Top GitHub Comments

1reaction

YohanSciubukgiancommented, Apr 6, 2020

Thanks for your answers. I confirm that the issue is relative to the combination of DISTINCT + (TOP x or OFFSET/LIMIT)

1. Results comparison using DISTINCT & TOP :

QUERY	MaxItemCount	SDK 3.6.0	SDK 3.7.0
SELECT DISTINCT TOP 5 c FROM c	null	5.12	133.86
SELECT DISTINCT TOP 5 c FROM c	-1	5.12	133.86
SELECT DISTINCT TOP 5 c FROM c	1	23.1	41.58
SELECT DISTINCT TOP 5 c FROM c	5	5.12	5.12
SELECT DISTINCT TOP 10 c FROM c	5	10.24	15.38
SELECT DISTINCT TOP 10 c FROM c	10	5.76	5.76

2. Results comparison using DISTINCT & OFFSET/LIMIT :

QUERY	Max Item Count	SDK 3.6.0	SDK 3.7.0
SELECT DISTINCT c FROM c OFFSET 0 LIMIT 5	null	5.12	133.86
SELECT DISTINCT c FROM c OFFSET 0 LIMIT 5	-1	5.12	133.86
SELECT DISTINCT c FROM c OFFSET 0 LIMIT 5	1	23.1	41.58
SELECT DISTINCT c FROM c OFFSET 0 LIMIT 5	5	5.12	5.12
SELECT DISTINCT c FROM c OFFSET 0 LIMIT 10	5	10.24	10.24
SELECT DISTINCT c FROM c OFFSET 5 LIMIT 10	5	15.38	25.66
SELECT DISTINCT c FROM c OFFSET 5 LIMIT 15	5	20.52	20.52
SELECT DISTINCT c FROM c OFFSET 0 LIMIT 5	10	5.12	5.76
SELECT DISTINCT c FROM c OFFSET 5 LIMIT 10	10	10.9	17.32
SELECT DISTINCT c FROM c OFFSET 5 LIMIT 15	10	11.54	17.32

@bchong95 The fix your are talking about is changing the behavior & usage of the SDK. Can it be adapted to match the fix you have done in 3.7.0 & to keep the SDK behaves as it was in previous versions ? Can we also add some tests in order to avoid any RequestCharge regression in the futur ?

0reactions

Mortana89commented, Apr 5, 2020

This seems like a very unintuitive approach which is quite confusing for SDK users. Can this be documented somewhere as the difference in RU is not really acceptable?