MaxItemCount and FeedIterator.HasMoreResult retrieved less items than expected.

Describe the bug We use QueryRequestOptions.MaxItemCount and FeedIterator.HasMoreResult to retrieve the specific number of items back like below sample code:

while ((requestOption.MaxItemCount <= 0 || list.Count < ro.MaxItemCount) && feedIterator.HasMoreResults)
                {
                    FeedResponse<T> feed = await feedIterator.ReadNextAsync();
                    foreach (var item in feed)
                    {
                        list.Add(item);
                    }
                }

This should retrieve the number of MaxItemCount back. It works for most cases and I test locally it also works welll.

But in PROD env for some cases, we only retrieve small amount of item results back when there are far more item result that meet the query filter condition that exist in the database. For our case, MaxItemCount is 150, item that meet the query filter condition is more than 50000, only retrieve 95 item back for lot of request.

To Reproduce Can’t reprouce locally.

Set MaxItemCount = 150, data that meet the query condition is more than this

Expected behavior Retrieve 150 items back

Actual behavior only less than 150 items are retrieved back.

Environment summary SDK Version: 3.31.0 OS Version Windows

[lubyrex]

@adityasa Thanks for the response. However, looking at the examples I fail to understand the difference between @blankor1’s code and the provided code sample in the link you provided.

Let me try to rephrase the question to make it clearer: I wrote a code snippet from the link you provided above and made some slight modifications:

            string queryText = "select * from c"; // cross partition query
            IList<Family> families1 = new List<Familiy>();
            using (FeedIterator<Family> query = container.GetItemQueryIterator<Family>(
                queryText,
                requestOptions: null)) // Setting request options null here
            {
                while (query.HasMoreResults)
                {
                    foreach (Family family in await query.ReadNextAsync())
                    {
                        families1.Add(family);
                    }
                }
            }
            
            IList<Family> families2 = new List<Familiy>();
            using (FeedIterator<Family> query = container.GetItemQueryIterator<Family>(
                queryText,
                requestOptions: new QueryRequestOptions  () {  MaxItemCount = 5 })) // Setting request options's MaxItemCount to some number
            {
                while (query.HasMoreResults)
                {
                    foreach (Family family in await query.ReadNextAsync())
                    {
                        families2.Add(family);
                    }
                }
            }

Questions:

1. Would families1 contain all the Familiy entities in the container given that the container has multiple partitions? Or do we need to manually manipulate response.ContinuationToken and call container.GetItemQueryIterator multiple times in order to iterate through all the Famility entities?
2. Would families1 and families2 lists contain the same results?

[adityasa]

Based on the (limited) code snippet above, the behavior is by design. If all results need to be fetched for a query, continuation token must be fully drained in the application code. Here’s a resource that explains this in cosmos db: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/query/pagination

Here’s a code sample in .NET sdk that shows how to do this: https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos.Samples/Usage/Queries/Program.cs#L294

Specifically, please note that pagination loop for the purposes of draining the continuation token needs to be implemented based on HasMoreResults property and rather should not depend on MaxItemCount, # of documents returned in the response etc.

Hope this helps.