conditional evaluation of nodes / sub-pipelines

Description

Is there a way to declare a pipeline that contains nodes (sub-pipelines) that are only evaluated based on the value of a predicate?

Context

Possible Implementation

Here’s conceptually what I would like to achieve:

node_if executes pipeline_a if the predicate is true and pipeline_b otherwise.

def condition(val):
    return val < 10


def do_a1(val):
    return f"a1-{val}"

def do_a2(val):
    return f"a2-{val}"


def do_b(val):
    return f"b-{val}"

pipeline_a = Pipeline([
    node(func=do_a1, inputs=['val'], outputs='val_a1', name='do_a1'),
    node(func=do_a2, inputs=['val_a1'], outputs='val_a2', name='do_a2')
])

pipeline_b = Pipeline([
    node(func=do_b, inputs=['val'], outputs='val_b', name='do_b'),
])

pipeline = Pipeline([
    node_if(func=condition, inputs=['val'], branch_true=pipeline_a, branch_false=pipeline_b),
])

Possible Alternatives

(Optional) Describe any alternative solutions or features you’ve considered.

[lorenabalan]

Hi @khdlim, thank you for sharing your experience, this is invaluable feedback!! Please see my inline responses, hoping they make sense to you. Feel free to reach out otherwise!

kedro-viz only allows to use /conf/base without an --env option, but that’s a separate issue

That’s a good point and I believe it has been raised before. We’re in the process of rethinking and improving the experience of KedroContext / ProjectContext and we’ll be sure to include this type of arguments in the discussion.

but I was wondering if the user was not designed to pass arguments to create_pipeline to begin with.

That is the right question to ask. If you look at the architecture diagram you can see that Pipeline and DataCatalog don’t know about each other. You do have access to the catalog inside create_pipeline, as it’s a property on the KedroContext class. While you can access it to get some information, we don’t recommend mutating it - weird things might happen as a result.

I also saw you mention in another post that separate pipelines would be available soon, that would be a godsend

You are in luck! We released version 0.15.2 yesterday, fetch it while it’s hot. 😃 You can check out a summary of the introduced changes here.

[lorenabalan]

I am closing this issue as answered, but please feel free to open a new one if there’s a need for further clarification. 😃