TimeWindowedPartitionConfigs should be possible to offset
See original GitHub issueThe documentation for the …_partitioned_config() functions states that the “start_date” parameter is “start_date (Union[datetime.datetime, str]) – The first date in the set of partitions. Can provide in either a datetime or string format.”
For weekly partitioned configs (and monthly) this is not really the case, as we use a 0 0 * * 0
cron schedule, and the start_date is only used to determine when that 0 0 * * 0
iterator will start.
This may be intended behavior, but there should be some functionality for being able to create weekly partition sets that end on different days of the week than Sunday (similar functionality should probably exist for all the other time windowed partition functions).
Issue Analytics
- State:
- Created 2 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
How shall we read the Kafka topics in a given time range?
The solution that I can think of is to first find out the largest offset for the beginning of the time range, and...
Read more >ConsumerRebalanceListener (kafka 2.4.0 API)
By saving offsets in the onPartitionsRevoked(Collection) call we can ensure that any time partition assignment changes the offset gets saved. Another use is ......
Read more >Configuring a Streams Application | Confluent Documentation
A host:port pair pointing to an embedded user defined endpoint that can be used for discovering the locations of state stores within a...
Read more >Windowing operations — pandas 1.5.2 documentation
If a time based offset is provided, the corresponding time based index must be monotonic.
Read more >Tables and Windowing — Faust 1.9.0 documentation
Iterating over keys/values/items in a windowed table. ... If the stream and table are not co-partitioned, we could end up with a table...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
PR is in review now!
PR is merged, will be included in this week’s release