question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Reinstate CheckpointAsync

See original GitHub issue

Is your feature request related to a problem? Please describe.

I’ve decoupled my consumption from my checkpointing (i.e., I don’t necessarily synchronously process all change feed items, instead letting reading get ahead of the checkpointable position in order that I can manage buffering and retries for performance)

Describe the solution you’d like

Provide an overload that exposes something akin to the IChangeFeedProcessorContext.CheckpointAsync method which the v2 CFP API formerly exposed (it’s present but internal atm)

Describe alternatives you’ve considered

Only alternative is to use the v2 CFP SDK, but that closes off tonnes of options and is not tenable from my perspective.

Additional context

  • I flagged this to @ealsur some time ago in the context of some other work in this repo. While https://github.com/jet/equinox/pull/144 illustrates that the V3 SDK provides some very nice cleanup in general for a relatively complex use case, not being able to port the CFP aspect easily presents a problem in the medium term with having client teams adopt the V3 SDK.

  • Consumption code

    • shows how we use (and need, CheckpointAsync)
    • illustrates that in general, the V3 API is good - it will cut out a lot of boilerplate wrapping we formerly had to provide.
    • never lets an exception escape from the processor observer function, avoiding the fact that the CFP logic does not retry and/or otherwise correctly handle exceptions.
    • illustrates the level of instrumentation (i.e. context re logging Range Ids etc) which cannot be exposed at the present time due to the absence of IChangeFeedProcessorContext (see related: https://github.com/Azure/azure-cosmos-dotnet-v3/issues/400)
  • Document parsing code

    • this code cannot be directly ported due to the fact that the Document class has become internal in V3 - it’d be nice to have a sample illustrate how one might most cheaply probe documents to determine whether they are parseable as a given type as was formerly possible.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:15 (12 by maintainers)

github_iconTop GitHub Comments

2reactions
ealsurcommented, Mar 19, 2020

Sadly it got a bit pushed back but prioritized still. We are working to release Change Feed pull support and this will be worked right after.

2reactions
ealsurcommented, Feb 11, 2020

This is coming back in V3, right now we got jumped by high-pri work. We want to have that and Context and Estimator with all leases for March.

Read more comments on GitHub >

github_iconTop Results From Across the Web

PartitionContext.CheckpointAsync Method
CheckpointAsync (). Writes the current offset and sequenceNumber to the checkpoint store via the checkpoint manager. CheckpointAsync(EventData).
Read more >
Save and load checkpoints - Flax - Read the Docs
Save and load checkpoints#. This guide demonstrates how to save and load Flax checkpoints with Orbax. Orbax provides a variety of features for...
Read more >
Monitoring Checkpointing | Apache Flink
Monitoring Checkpointing # Overview # Flink's web interface provides a tab to monitor the checkpoints of jobs. These stats are also available after...
Read more >
tf.train.CheckpointOptions | TensorFlow v2.13.0
Checkpoint.restore() methods to adjust how variables are saved/restored. Example: Run IO ops on "localhost" while saving a checkpoint:.
Read more >
Distributed Deep Learning with Ray Train User Guide
Distributed Deep Learning with Ray Train User Guide#. This guide explains how to use Train to scale PyTorch, TensorFlow and Horovod. In this...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found