[AIP-31] Create XComArg model

Description

XComArg is a class that references an XCom entry from an operator. Can be used to explicitly set XCom dependencies and explictly pass results between operators.

Class attributes:

• operator: BaseOperator: Origin operator instance.
• key: str: Stores key for XCom value. Defaults to airflow.models.xcom. XCOM_RETURN_KEY

Class methods

• get(context: dict)-> Any: Resolves XCom value given operator and key.
• __getitem__(key, str) -> XComArg: Easy method to create new XComArg using the same operator but a different key.
• add_downstream_dependent(operator: BaseOperator)->None: Add an operator as a downstream dependency of operator. [Optional] Makes this task simpler.

Proposed implementation: https://github.com/casassg/corrent/blob/master/corrent/xcom_arg.py

Use case / motivation

• Explicit way to pass around results between operators.
• This object is a reference to an XCom value that has not been created and will need to be resolved in the future.
• It can generate new XComArgs that point to the same operator but has a different key by using the key accessor. E.g: output[‘train’].
• To do so, it includes as class attributes both the originating operator instance and a key (both needed to pull the XCom).
• Has a property op that allows access to the origin operator.

Stretch goal

• Store type class for the XCom and lie to MyPy when trying to check class signature.

[jonathanshir]

Hey guys, the databand team and I decided to implement [AIP-31] starting tomorrow. Will be in touch with you soon for implementation discussions. Thanks, Johnny 😃

[evgenyshulman]

it’s definitely an option for people who wants a “purely function” approach from the beginning. Most of Airflow users are already very invested in the current implementation without functional operators.

1. They will have to rewrite their current implementation ( extract into function with @task).
2. In case it’s part of internal operator implementation it’s even harder to do
3. It will affect an execution graph (extra node), while some Airflow users are trying to keep graph very clean.

IMHO: ( as I wrote at current PR), we should decide before we push implementation into Airflow/master and we should base our decision on some examples we wrote with end to end usage of AIP-31 implementation. I see a point in keeping everything as simple as possible, at the same time, I want all users to be happy if it doesn’t affect implementation in a bad way 😃