Dynamically change task priorities

Dask is great for the current workflow that I’m using - read a number of tables, run some functions on each, output the result as a table where each cell either shows the initial table ID or the result of one of the functions in the pipeline.

Now that I’m implementing a UI for this table, the need to sort and filter interactively it becomes apparent, as well as the need to show intermediate results right as they’re ready.

Judging by the source of distributed.scheduler, the current implementation doesn’t allow for dynamic task priority. Are there any plans to implement it? Or maybe any ideas on how I can implement it in the best way possible?

[mrocklin]

We would probably just send an update-graph message with new priority values without any graph. This almost certainly doesn’t work now, but could be made to.

The real challenge here is that the scheduler currently auto-increments priority on every new update_graph call, preferring older tasks to newer ones. It’s not clear to me how we would change this policy. Maybe there is a third policy that overrides the other two.

(user-defined-priority, scheduler first come first served priority, priority from graph placement)

[jakirkham]

Opened issue ( https://github.com/dask/distributed/issues/1753 ), which is similar to this one except that it would auto-propagate priority changes via user operations on Dask collections that have Futures in them.