Accumulate task inputs

Use Case

Please provide a use case to help us understand your request in context

It would be great to be able to accumulate multiple task calls and execute them all at once. Examples are neural networks and database queries: doing one request at a time we don’t use resources efficiently and get ourselves a bottleneck.

Solution

Please describe your ideal solution

The simplest solution is to implement something like this.

We need to add flush_every and flush_interval parameters to the Task class.

Alternatives

Please describe any alternatives you’ve considered, even if you’ve dismissed them

I’ve read about Task Loops, but they don’t quite work because they just run the task with the same arguments multiple times, whereas we need to run job once, but with multiple arguments.

[jcrist]

Since prefect stores state (and if using cloud/server also posts state to a server) for every task, there is always going to be overhead per task. If you’re running fast tasks in prefect, you might consider wrapping these into a bulk api and having prefect schedule them at a higher level:

def my_fast_task(x):
    ...

def batched_fast_task(args):
    return [my_fast_task(x) for x in args]

We could implement batching in prefect, but that gets complicated to schedule when dealing with retries, etc… IMO I’d like to see more use cases that require changes (and can’t be done with user-level batching) before going down this route.

For mapped tasks there’s the suggestion in #2459 that I think we would be willing to support, would this be sufficient for your needs?

[github-actions[bot]]

This issue was closed because it has been stale for 14 days with no activity. If this issue is important or you have more to add feel free to re-open it.