Query performance issues between 3.9.0 and 3.17.0

Describe the bug We’re upgrading from 3.9.0 to 3.17.0-preview on a performance intensive product. We’ve noticed a 23 ms per call increase in query times between the 2 SDKs.

To Reproduce Upgraded from 3.9.0 to 3.17.0-preview. Read the release notes on all releases to follow the upgrade notes. Added the following client performance items: ` .WithConnectionModeDirect()

options.PortReuseMode = PortReuseMode.PrivatePortPool; options.IdleTcpConnectionTimeout = TimeSpan.FromHours(1); options.AllowBulkExecution = false; `

Also converted query code from:

` query.ToStreamIterator();

to container.GetItemQueryStreamIterator(query.ToQueryDefinition(), null, new QueryRequestOptions { MaxConcurrency = -1, MaxBufferedItemCount = -1 })) `

Expected behavior Performance difference between the two versions to be nominal.

Actual behavior Ran load tests and observed query durations take 20 ms longer with newer APIs and changes.

Environment summary SDK Version: 3.17.0-preview OS Version Linux containers

Additional context Below is a graph of performance comparisons:

Yellow box is the 3.17.0-preview library upgrade with no code changes, Green box is 3.9.0 pre-upgrade Red box is 3.17.0-preview with performance related code changes made.

[dpiessens]

@timsander1

The queries are different but the output doc size is about the same:

3.9: Query had OutputDocumentCount=18;outputDocumentSize=11511 3.17: Query had Output Document Count =16, Output Document Size: 10, 238 bytes

The query is actually the same, the results have to do with the process it’s embedded in and the comparison. 3.17 ran first so it was loading the data. When it ran the query the record it was working on wasn’t in the DB yet so not in the results. The target being loaded also had another record after this one so that was loaded. 3.9 was then run so it was re-loading the results. This affects the upsert and replace operations but those had no performance changes. Queries for both loads are identical aside from the 2 additional records, but I don’t this this can account for the difference.

Our process tracks the load, origin record, method name and the query. I made all of these items the key when I lined up comparisons so I’m completely confident the results are from the same query with the same parameters.

@j82w

1. Yes we use a CI process to construct each container image, and both tests were deployed to the same AKS cluster. That cluster had no other work on it during the tests and the container has an anti-affinity so they are distributed on the same node set.
2. Yes both the cluster an DB are in East US 2
3. Yes I can provide additional traces. I’ll send them to @timsander1 as they will have information I’m not comfortable publicly posting.

[timsander1]

Hi @dpiessens

Would it be possible to send two traces for the same query with the same filter? The query metrics are different enough that it’s hard to rule this out as a possible reason for the difference.

Thanks!