question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Change feed pull model - High request charges

See original GitHub issue

Describe the bug Hi, I’m not sure if this is a bug or expected behavior but here it is.

I’m trying to use the change feed pull model to query all documents in a specific partition key and save the continuation token when there is nothing left to read. In the next run, I will be able to use the saved continuation token to continue querying the change feed for new changes. The thing is, I’m not sure if the request charge should be that high. For only 141 documents in the collection, I see request charges from 10 000 to almost 25 000. See below.

var changeFeedIterator = _eventStoreContainer.GetChangeFeedIterator<EventStoreEvent<object>>(
    ChangeFeedStartFrom.Beginning(FeedRange.FromPartitionKey(new PartitionKey(instance.Id))),
    ChangeFeedMode.Incremental
);

while (changeFeedIterator.HasMoreResults)
{
    var response = await changeFeedIterator.ReadNextAsync();
    if (response.StatusCode == HttpStatusCode.NotModified)
    {
        Console.WriteLine($"No new changes");
        break;
    }
    eventsToReplay.AddRange(response);
}

image image

I tested with a simple LINQ query and the request charge was only 11.39 to query the same 141 documents

var feedIterator = _eventStoreContainer.GetItemLinqQueryable<EventStoreEvent<object>>(requestOptions: requestOptions)
    .ToFeedIterator();

while (feedIterator.HasMoreResults)
{
    var response = await feedIterator.ReadNextAsync();
    eventsToReplay.AddRange(response);
}

image

So is this normal? I’m expecting a higher request charge because it needs to read all the changes in the partition but should it be that high?

Environment summary .net core 3.1 “Microsoft.Azure.Cosmos” Version=“3.24.0” Windows 11

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

2reactions
jriouxrandstadcacommented, Feb 8, 2022

Here are the diagnostics logs diagnostics.zip It contains the logs for the change feed iterator call and the LINQ queryable call Thank you

0reactions
jriouxrandstadcacommented, Feb 8, 2022

Thank you for the information.

The thing with query ORDER by _ts is I cannot continue where I was on the next function run as I do with the ChangeFeedIterator continuation token. The idea was to use the ChangeFeedIterator with a continuation token to have an on-demand event processing from an event store. I cannot use the _ts as multiple events can have the same _ts and we don’t want to process the same event twice.

Is there an estimated date/planned fix for this?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Change feed pull model in Azure Cosmos DB
In the change feed pull model, you can use the FeedRange to parallelize the processing of the change feed. A FeedRange represents a...
Read more >
CosmosDB Change Feed Processing. Introduction
Change Feed reads have an RU charge, so funnelling all the change feed request to a secondary region means that the additional read...
Read more >
Day 28 - Event Sourcing with Azure Cosmos DB Change Feed ...
Help the elves track every event that happens this holiday season. The change feed in Azure Cosmos DB is one of the most...
Read more >
First steps of the Change Feed pull model in Azure ...
The beauty of Change Feed is that it can read data in the order in which it is updated, as long as the...
Read more >
Causes of Inflation | Explainer | Education
This Explainer describes the main causes of changes in the inflation rate. ... Image showing the three main causes of inflation: demand-pull, cost-push...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found