ReadNextAsync goes crazy and load all documents from collection, against executing query.

Hi everyone. I have nasty case for you, which we observe on our production code. It’s random, it’s crazy and we are out of ideas why this happening. We need help from you as it seems to have source inside SDK itself. Take a seat and enjoy.

1. Executed query: SELECT TOP 10 VALUE root FROM root WHERE ((true OR (root["User"]["IsDeleted"] = false)) AND StringEquals(root["User"]["AllegroData"]["Email"], "[e-mail]", true)) ORDER BY root["_ts"] DESC

2. Executed in portal:

3. Executed on our production:

FeedIterator.ToListAsync: PageCount: “1”, RuUsage: “1971.7099999999998”, PageNumber: 0 FeedIterator.ToListAsync: PageCount: 0, RuUsage: 46456.38, PageNumber: 1 FeedIterator make more than one roundtrip! RuUsage: 48428.09, Query: “query from point 1”

So, one ReadNextAsync call, produce most of this dependencies, resulting with 46k RuUsage.

1. This wrong execution, has sum of dependencies on Metric-retrievedDocumentCount equal documents count in collection - it means, all documents from collection are loaded on cosmo server. On other hand, Metric-outputDocumentCount, has always one or zero results (as expected from query)

2. This happens randomly, couple times at day, always for the same query, but different parameters.

3. It takes time (see picture from point 2)

4. SDK versions Microsoft.Azure.Cosmos (3.16) Microsoft.Azure.DocumentDB.Core (2.13.1)

5. ToListAsync code:

public static async Task<List<T>> ToListAsync<T>(this FeedIterator<T> query, ILogger logger = null, string queryString = null)
        {
            var results = new List<T>();
            double ruUsage = 0;
            var count = 0;
            string firstPageCount = "";
            string firstRu = "";
            while (query.HasMoreResults)
            {
                var page = await query.ReadNextAsync();
                results.AddRange(page);
                ruUsage += page.RequestCharge;
                if (count == 0)
                {
                    firstPageCount = page.Count.ToString();
                    firstRu = page.RequestCharge.ToString(CultureInfo.InvariantCulture);
                }
                if (count > 0)
                {
                    if (count == 1)
                    {
                        logger?.Information(
                            "FeedIterator.ToListAsync: PageCount: {pageCount}, RuUsage: {ruUsage}, PageNumber: {count}",
                            firstPageCount, firstRu, 0);
                    }
                    logger?.Information(
                        "FeedIterator.ToListAsync: PageCount: {pageCount}, RuUsage: {ruUsage}, PageNumber: {count}",
                        page.Count, page.RequestCharge, count);
                }
                count++;
            }

            if (count > 1)
            {
                logger?.Warning("FeedIterator make more than one roundtrip! RuUsage: {ruUsage}, Query: {queryString}", ruUsage, queryString);
            }
            
            return results;
        }

Do you have some ideas why this happening? It is possible that something is wrong with StringEquals function? Some edge case where it doesn’t work properly?

[neildsh]

This should already be fixed by https://github.com/Azure/azure-cosmos-dotnet-v3/pull/2168

[bartelink]

@j82w Thanks. Just to be clear: my ‘report’ should be considered wild speculation for now.

I’m not currently running the ingestion process so can’t even chart memory growth atm.

The other thing that’s impeding my progress toward doing proper issue reporting is that I’m having to mix V2 CFP usage with usage of the V3 client as we wait on https://github.com/Azure/azure-cosmos-dotnet-v3/issues/1765