Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

hoodie.cleaner.commits.retained max value?

See original GitHub issue

Is there a greatest value possible for the number of commits the hudi cleaner can retain?
What about the greatest possible value for hoodie.keep.min.commits and hoodie.keep.max.commits?
What is the difference between hoodie.cleaner.commits.retained and hoodie.keep.min.commits? If I understand correctly, the cleaner commits retained is the maximum number of commits we can query incrementally/point in time from, keep min commits is the least number of commits stored (for example in S3) and keep max commits is the maximum number of commits stored.
Are any commits between the keep min commits and keep max commits thresholds archived as log files?
Are any commits that exceed the max commit threshold deleted?
Are there performance issues if I set these values very large? For example:

hoodie.cleaner.commits.retained: 20000, hoodie.keep.min.commits: 20001, hoodie.keep.max.commits: 30000

Thanks in advance

Issue Analytics

State:
Created 2 years ago
Comments:5 (3 by maintainers)

Top GitHub Comments

1reaction

moritzmeistercommented, Jul 4, 2022

This thread is very helpful, this should end up in the documentation 😃

0reactions

n3nashcommented, Jun 4, 2021

Yes, that is correct @AkshayChan. Closing this ticket, please open one if you need further assistance

Top Results From Across the Web

All Configurations | Apache Hudi

Comma separated list of file paths to read within a Hudi table. Default Value: N/A (Required) Config Param: READ_PATHS. hoodie.datasource.read.incr.

[GitHub] [hudi] n3nash edited a comment on issue #2841: hoodie ...

Is there a greatest value possible for the number of commits the hudi cleaner can retain? The num_commits is eventually converted to an...

Compaction and Cleaning Configurations

hoodie.compact.inline.max.delta.commits. Specifies the maximum number of delta commits to be retained before inline compression is triggered. 5 ; hoodie.

Constant Field Values (hudi-client 0.5.3 API) - javadoc.io

... "hoodie.cleaner.commits.retained". public static final String · CLEANER_FILE_VERSIONS_RETAINED_PROP, "hoodie.cleaner.fileversions.retained".

HudiJob.py - AWS

Again, if the user overwrites the config with any value other than 'MoR' we will create CoW ... 'hoodie.cleaner.commits.retained': 10} initLoadConfig ...