Support for composite tasks

Current behavior

Currently, it appears there is no way to compose tasks.

Proposed behavior

I think this is a good example of where composite tasks would be useful:

@task
def read_table(path):
  ... # Load some tabular data structure

@task 
def get_merged_tables(path1, path2):
  t1 = read_table(path1)
  t2 = read_table(path2)
  return t1.join(t2)

Both read_csv and get_merged_tables are useful tasks on their own in this case, likely called individually from several different places in a pipeline. Is there a better workaround than the following or anything on the roadmap to this end?

def _read_table(path):
  ...

@task 
def read_table(path):
  return _read_table(path)

@task 
def get_merged_tables(path1, path2):
  t1 = _read_table(path1)
  t2 = _read_table(path2)
  return t1.join(t2)

[eric-czech]

Yep, makes sense. Calling .run seems reasonable for the former but I’m much more interested in the latter (nested dag structure). I don’t see any kind of extra computational expressivity that comes from it – I was only thinking that it’s a practical feature for big pipelines especially since one of my main interests in Prefect is the stuff that makes it easier to create tasks at a granular level. Finding the right balance is a little difficult IMO and I find myself reformulating tasks sometimes rather than keeping the conceptual model I’d like to have in order to fit the flat-dag layout.

[jcrist]

Got it. We’re tracking “subflows” (what we’re calling this kind of feature) in #1745, you can follow along there to see the conversation. Closing.