question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

TimeWindowedPartitionConfigs should be possible to offset

See original GitHub issue

The documentation for the …_partitioned_config() functions states that the “start_date” parameter is “start_date (Union[datetime.datetime, str]) – The first date in the set of partitions. Can provide in either a datetime or string format.”

For weekly partitioned configs (and monthly) this is not really the case, as we use a 0 0 * * 0 cron schedule, and the start_date is only used to determine when that 0 0 * * 0 iterator will start.

This may be intended behavior, but there should be some functionality for being able to create weekly partition sets that end on different days of the week than Sunday (similar functionality should probably exist for all the other time windowed partition functions).

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
jamiedemariacommented, Mar 24, 2022

PR is in review now!

0reactions
jamiedemariacommented, Mar 28, 2022

PR is merged, will be included in this week’s release

Read more comments on GitHub >

github_iconTop Results From Across the Web

How shall we read the Kafka topics in a given time range?
The solution that I can think of is to first find out the largest offset for the beginning of the time range, and...
Read more >
ConsumerRebalanceListener (kafka 2.4.0 API)
By saving offsets in the onPartitionsRevoked(Collection) call we can ensure that any time partition assignment changes the offset gets saved. Another use is ......
Read more >
Configuring a Streams Application | Confluent Documentation
A host:port pair pointing to an embedded user defined endpoint that can be used for discovering the locations of state stores within a...
Read more >
Windowing operations — pandas 1.5.2 documentation
If a time based offset is provided, the corresponding time based index must be monotonic.
Read more >
Tables and Windowing — Faust 1.9.0 documentation
Iterating over keys/values/items in a windowed table. ... If the stream and table are not co-partitioned, we could end up with a table...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found