Solids are skipped when their optional inputs are skipped

Given a “fan in” solid with N fans, if one of the N fans is an input from an upstream solid that gets skipped, the fan in solid gets skipped too, even though that skipped input was marked as Optional.

    A
   / \
  B   C
   \ /
    D
    |
    E

D defines the input from C as optional, but if C is skipped then D and E won’t run run.

Some code to show how this issue is biting us in real life:

def construct_fan_in_solid(fan_count):

    assert fan_count > 1

    solid_name = f'fan_in_{fan_count}'

    input_defs = [ InputDefinition('fan_1', Any) ]
    for fan_id in range(2, fan_count + 1):
        input_defs.append(InputDefinition(f'fan_{fan_id}', Optional[Any], default_value=None))

    def _fan_in(context, *fans):
        yield Output(fans[0], 'fan_1')

    return SolidDefinition(
        name=solid_name,
        input_defs=input_defs,
        output_defs=[
            OutputDefinition(name='fan_1', dagster_type=Any)
        ],
        compute_fn=_fan_in,
    )

fan_in_11 = construct_fan_in_solid(11)

(Aside: I created this solid using a factory because of #1817. First attempt used a List, but it wasn’t clear if the problem was the issue described here or if Dagster was misunderstanding what Optional[List[Optional[Any]]] means.)

Dagit correctly recognizes that all fans except the first are optional:

The above is used like this:

@pipeline
def my_pipeline():
    main_id, other_id = solid2(solid1()) # other_id is an optional output
    fan1 = solid3(main_id)
    fan2 = solid4(other_id)
    ...
    main_id = fan_in_11(main_id, fan1, fan2, ...)
    solid5(main_id)

The problem: if other_id is not outputted by solid2, fan_in_11 doesn’t run because Dagster complains its dependencies (fan2 from solid4) didn’t produce an output - despite their being marked optional inputs.

Dagster v0.8.5

[alangenfeld]

Working on a fix for how the skipping behavior proceeds through fan-in dependencies https://dagster.phacility.com/D3767, hoping to have it in for the weekly release.

Note that the Optional typing for inputs, follows the python type meaning of Union[X, None] https://docs.python.org/3/library/typing.html#typing.Optional so just specifies that None is an acceptable value as opposed to interacting with the skipping behavior of is_required=False outputs. That said, the fact this is so confusing is noted and we are thinking of how we can improve.

[alangenfeld]

I think we can close this one.