Allow consumption of Change Feed as a pull model
See original GitHub issueIs your feature request related to a problem? Please describe. We are using the ChangeFeedProcessorBuilder in our WebAPI to update an in-memory cache with recent changes to a container as outlined in the change feed use cases documentation. We have multiple instances of our WebAPI that each have their own cache so each instance is working on a single processor and doesn’t need to share updates.
Describe the solution you’d like
Allow us to bypass the need for a lease container to read from the change feed. One solution might be to make ChangeFeedProcessorBuilder.WithInMemoryLeaseContainer()
public with a way of specifying a starting continuation Token if desired.
Another would be to expose a container method that allows us to make a direct request for the change feed with our own continuation token and leave it to the caller to write the timer interval and storage mechanism (if desired).
Describe alternatives you’ve considered
Giving a unique processor name per WebAPI does work but it is unnecessarily creating lease documents and updating them often (we are seeing ~22 requests per minute on our lease container per WebAPI. This multiplies based on number of logical partitions). WithLeaseConfiguration()
seems to allow overriding the defaults to be much higher but there is no guidance for what appropriate values would be.
Additional context
Failing to provide a WithLeaseContainer()
call on the ChangeFeedProcessorBuilder results in an exception saying to add it or use WithInMemoryLeaseContainer
which is not accessible.
Issue Analytics
- State:
- Created 4 years ago
- Reactions:1
- Comments:16 (8 by maintainers)
Top GitHub Comments
@WimVergouwe please see linked PR
In-memory would work like the current CFP, either start from the beginning, or start at some point in time, or from now on. No way to pass any other state.