Continuation Token Exception After Using Skip()

We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.

Describe the bug Continuation token exception (MalformedContinuationTokenException)

To Reproduce

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Bogus;
using Microsoft.Azure.Cosmos;
using Microsoft.Azure.Cosmos.Linq;
using Newtonsoft.Json;
using static System.Console;

var client = new CosmosClient(Environment.GetEnvironmentVariable("ConnectionString"));
var database = client.GetDatabase("paging-db");
var response = await database.CreateContainerIfNotExistsAsync(new ContainerProperties
{
    Id = "people",
    PartitionKeyPath = "/id"
});

int pageSize = 10;
int page = 4;

var options = new QueryRequestOptions {MaxItemCount = pageSize};
string continuationToken = null;


var container = response.Container;

int itemsToSkip = (pageSize * page) - pageSize;

var query = container
    .GetItemLinqQueryable<Person>(false, continuationToken, options)
    .Where(i => i.Age > 5)
    .Skip(itemsToSkip);


int count = await container
    .GetItemLinqQueryable<Person>(false, continuationToken, options)
    .Where(i => i.Age > 5).CountAsync();

WriteLine($"{count} items");

using (var iterator = query.ToFeedIterator())
{
    var feedResponse = await iterator.ReadNextAsync();

    WriteLine($"Page {page} returned {feedResponse.Resource.Count()} results and cost {feedResponse.RequestCharge} RUs");
    WriteLine($"Names in page [{string.Join(",", feedResponse.Resource.Select(i => $"{i.Name} ({i.Age})"))}]");

    continuationToken = feedResponse.ContinuationToken;
    page++;
    
};

query = container
    .GetItemLinqQueryable<Person>(false, continuationToken, options)
    .Where(i => i.Age > 5);

using (var iterator = query.ToFeedIterator())
{
    var feedResponse = await iterator.ReadNextAsync();

    WriteLine($"Page {page} returned {feedResponse.Resource.Count()} results and cost {feedResponse.RequestCharge} RUs");
    WriteLine($"Names in page [{string.Join(",", feedResponse.Resource.Select(i => $"{i.Name} ({i.Age})"))}]");

    continuationToken = feedResponse.ContinuationToken;
    page++;
};

async Task Seed()
{
    Faker<Person> peopleFaker = new();
    peopleFaker
        .RuleFor(p => p.Name, f => f.Name.FullName())
        .RuleFor(p => p.Age, f => f.Random.Number(15, 45));

    List<Person> people = peopleFaker.Generate(100);
    foreach (var person in people)
    {
        await container.CreateItemAsync(person);
    }
}

public class Person
{
    [JsonProperty("name")]
    public string Name { get; set; }

    [JsonProperty("age")]
    public int Age { get; set; }
}

Expected behavior I expected the token to leave me back on the next page where I left off from the offset query.

Actual behavior A exception is thrown CosmosException 400 bad request saying the token is malformed.

Environment summary SDK Version: <PackageReference Include="Microsoft.Azure.Cosmos" Version="3.20.1" /> OS Version: MacOS & Windows

Additional context This has been tested against the real cosmos instance.

I have tested this on netcoreapp3.1 and net5.0.

This may be not be a use for tokens here. Any help would be appreciated.

[mumby0168]

I have come across this issue as part of working on this PR for the CosmosRepository implementation it has more details about the issue and how I plan to support paging: https://github.com/IEvangelist/azure-cosmos-dotnet-repository/pull/82

[mumby0168]

@mumby0168 the code has a bug. You cannot create a continuation token with one query and then attempt to use it with a different query. You must use the same query and settings that were used to create the continuation token.

See corrected code below, which also includes a less error prone alternative:

            int pageSize = 10;
            int page = 4;

            QueryRequestOptions options = new QueryRequestOptions { MaxItemCount = pageSize };
            string continuationToken = null;


            int itemsToSkip = (pageSize * page) - pageSize;

            IQueryable<Person> query = container
                .GetItemLinqQueryable<Person>(false, continuationToken, options)
                .Where(i => i.Age > 5)
                .Skip(itemsToSkip);

            int count = await container
                .GetItemLinqQueryable<Person>(false, continuationToken, options)
                .Where(i => i.Age > 5).CountAsync();

            Trace.WriteLine($"{count} items");

            using (FeedIterator<Person> iterator = query.ToFeedIterator())
            {
                FeedResponse<Person> feedResponse = await iterator.ReadNextAsync();

                Trace.WriteLine($"Page {page} returned {feedResponse.Resource.Count()} results and cost {feedResponse.RequestCharge} RUs");
                Trace.WriteLine($"Names in page [{string.Join(",", feedResponse.Resource.Select(i => $"{i.Name} ({i.Age})"))}]");

                continuationToken = feedResponse.ContinuationToken;
                page++;
            };

            QueryDefinition queryDefinition = query.ToQueryDefinition<Person>();
            using (FeedIterator<Person> iterator = container.GetItemQueryIterator<Person>(queryDefinition, continuationToken, options))
            {
                FeedResponse<Person> feedResponse = await iterator.ReadNextAsync();

                Trace.WriteLine($"Page {page} returned {feedResponse.Resource.Count()} results and cost {feedResponse.RequestCharge} RUs");
                Trace.WriteLine($"Names in page [{string.Join(",", feedResponse.Resource.Select(i => $"{i.Name} ({i.Age})"))}]");

                continuationToken = feedResponse.ContinuationToken;
                page++;
            };

            IQueryable<Person> query2 = container
                .GetItemLinqQueryable<Person>(false, continuationToken, options)
                .Where(i => i.Age > 5)
                .Skip(itemsToSkip);

            using (FeedIterator<Person> iterator = query2.ToFeedIterator<Person>())
            {
                {
                    FeedResponse<Person> feedResponse = await iterator.ReadNextAsync();

                    Trace.WriteLine($"Page {page} returned {feedResponse.Resource.Count()} results and cost {feedResponse.RequestCharge} RUs");
                    Trace.WriteLine($"Names in page [{string.Join(",", feedResponse.Resource.Select(i => $"{i.Name} ({i.Age})"))}]");

                    continuationToken = feedResponse.ContinuationToken;
                    page++;
                }
            }

Okay that makes sense thank you for looking at that, I have misunderstood how this works, not quite sure how we will handle this in our library will have to re-think.

Thanks for that 👍