Using static type checking as qa on a Flow library

Current behavior

As a user developing Flows, I struggle with debugging flows, since each of the tasks represents delayed execution. Running a flow is the most complete test, but since flows can change the state of external systems, it would be helpful to have some qa that my flow is set-up correctly before running/registering. This is especially impactful in the situation where I have a flow library and want to implement this qa as part of a CI pipeline.

Proposed behavior

For flows with Tasks that have data dependencies between them, I could qa on the types passed between different tasks. Right now, tasks are typed to hint at the types encountered when the flow is registered. Imagine a scenario, where the types hinted at the task execution result (the result of the delayed execution). You could run mypy and ensure that the dependencies between flows have sensible types (plus your IDE will be more helpful)

Example

from typing import Any, Callable, TypeVar

import prefect
import prefect.tasks.core.function


F = TypeVar('F', bound=Callable[..., Any])


def typed_task(fn: F, **task_init_kwargs: Any) -> F:
    """Same dynamic code as `prefect.task`, but preserves function signature for debugging in a Flow context.
    """
    return prefect.tasks.core.function.FunctionTask(fn=fn, **task_init_kwargs)  # type: ignore


@typed_task
def task_one() -> str:
    return "1"


@typed_task
def task_two(arg: str) -> int:
    return int(arg)


with prefect.Flow("test_type_hints") as flow:
    result_one: str = task_one()  # typed to hint at the execution result 
    result_two: int = task_two(result_one)
flow.register()

[dyllamt]

Update: It won’t be possible to support this until PEP 612 and changes to mypy go live https://github.com/python/mypy/issues/8645

[github-actions[bot]]

This issue was closed because it has been stale for 14 days with no activity. If this issue is important or you have more to add feel free to re-open it.