Support easily running part of a flow

Archived from the Prefect Public Slack Community

ben.fogelson: Is there an easy way to run part of a flow? I.e. run up to and including task x?

laura: Hi! No, I don’t think so without changing your flow code (for example simply commenting out whatever after x you had added to your flow graph). You can run individual tasks (re: https://docs.prefect.io/core/advanced_tutorials/task-guide.html#calling-your-task-s-run-method) but if your flow is complex that isn’t realistic to simulate a partial flow. It is an interesting idea though and I did a quick search and we don’t have an issue already open on it, so I can publish this thread into a new issue to request it!

ben.fogelson: Thanks <@UTHEQ4F9R>, that would be great!

laura: <@ULVA73B9P> archive “Support easily running part of a flow”

[jlowin]

We once had start_tasks as an option which ran a “pruned” subgraph of the flow (removed in #668). While the implementation itself was straightforward (we included all nodes downstream of the root nodes) the complexities it introduced were significant - users often were unaware that states upstream of the root tasks had to be properly passed in order for triggers/data dependencies to work as expected, and it resulted in some confusion. I think it’s definitely doable but the UX needs some work.

[madkinsz]

Hey @fjord-prefect – I think this will mostly be resolved by planned improvements to our caching interface. Basically, the upstream tasks that you don’t want to re-run will be easily set to run once per time period. You can actually do this now using cache_validators – https://docs.prefect.io/core/concepts/persistence.html#output-caching but it’s not as easy as we’d like.