[feature] Condition combinators for dsl.Condition

Feature Area

/area backend

What feature would you like to see?

Right now, dsl.Condition only allows to use a single comparison:

with dsl.Condition(param1 == value1):
  ...

Because ConditionOperator doesn’t define any custom __bool__ which would produce a warning, the following conditions silently work in unexpected ways:

with dsl.Condition(param1 == value1 and param2 == value2):  # yields param2 == value2
  ...

with dsl.Condition(param1 == value1  or param2 == value2):  # yields param1 == value1
  ...

Because and and or Python cannot be overloaded, the only thing one can do to prevent those misleading behaviour is to add a warning to __bool__ method of ConditionOperator (which would require turning it into an attr class instead of a namedtuple).

The same kind of a problem does occur in e.g. pandas, and cannot easily solved in Python. However, & (via __and__) and a custom method (e.g. and_then) are fine to define on ConditionOperator:

with dsl.Condition((param1 == value1) | (param2 == value2)):
  ...

with dsl.Condition((param1 == value1).or_else(param2 == value2)):
  ...

As seen above, brackets are needed, due to operator precedence (no overwritable binary operator has lower precedence than ==). Arguably, it’s still well readable and also enables complex conditions.

The generated yaml in such a case would include:

"$tasks.some-task.results.param1" == "value1" || "$tasks.some-task.results.param2" == "value2"

For more complex scenarios, braces would need to be used:

with dsl.Condition( ((param1 == value1) | (param2 == value2)) & (param3 == value3) ):
  ...

("$tasks.some-task.results.param1" == "value1" || "$tasks.some-task.results.param2" == "value2") &&
"$tasks.some-task.results.param3" == "value3"

For readability, if multiple operators of the same kind are used, those should not produce braces:

with dsl.Condition( (param1 == value1) | (param2 == value2) | (param3 == value3) ):
  ...

"$tasks.some-task.results.param1" == "value1" || "$tasks.some-task.results.param2" == "value2" ||
"$tasks.some-task.results.param3" == "value3"

Note: considering how Argo is using govaluate, handling and/or shouldn’t be a problem — __and__ (&) would compile to && and __or__ (|) would compile to ||. The ConditionOperator would be treated as a binary tree with non-ConditionOperator values at its leaves.

What is the use case or pain point?

This feature allows users to create much more complex conditions, combining multiple simple predicates. Typical use case is error handling.

Conjunction of conditions can be used e.g. if some handling (like logging) depends both on some status from component and the logging level defined in a pipeline parameter:

@dsl.pipeline(name="test error handling")
def test_error_handling(log_errors_of_level: int = 3):
  op_lvl_2 = dsl.ContainerOp(...)
  with dsl.Condition( (op_lvl_2.errCode != "") & (log_errors_of_level <= 2) ):
    LogErrorOp(op_lvl_2.errCode)

Alternative of predicates can be used to execute error-handling code if any of parallel actions failed (or produced some undesired result):

@dsl.pipeline(name="test error handling")
def test_error_handling(log_errors_of_level: int = 3):
  op1 = dsl.ContainerOp(...)
  op2 = dsl.ContainerOp(...)
  op3 = dsl.ContainerOp(...)
  with dsl.Condition( (op1.errCode != "") | (op2.errCode != "") | (op3.errCode != "") ):
    LogErrorOp(op1.errCode, op2.errCode, op3.errCode)

Is there a workaround currently?

While one can emulate “and” using hierarchy of conditions, although with poor readability both in DSL and generated yaml:

with dsl.Condition(param1 == value1):
  with dsl.Condition(param2 == value2):
    ...

…the same cannot be done with “or”.

---

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[maganaluis]

/reopen

[Tomcli]

related issue: https://github.com/kubeflow/pipelines/issues/482