Make runtime output metadata available to downstream ops

Requests for this:

• https://github.com/dagster-io/dagster/discussions/6913
• https://dagster.slack.com/archives/C01U954MEER/p1655806363783739
• https://dagster.slack.com/archives/C01U954MEER/p1655606789374799
• https://dagster.slack.com/archives/C01U954MEER/p1660902824466749
• https://dagster.slack.com/archives/C02NA9CUPU0/p1661200431642469

Workaround for now, if you’re using software-defined assets and you want to access it in the IO manager:

def handle_output(self, context):
    context.add_output_metadata({"foo": "bar"})

def load_input(self, context):
    asset_key = context.asset_key
    event_log_entry = context.step_context.instance.get_latest_materialization_event(asset_key)
    metadata = event_log_entry.dagster_event.event_specific_data.materialization.metadata
    foo_value = metadata["foo"].value
    assert foo_entry.text == "bar"

Workaround for now, if you’re using software-defined assets and you want to access it in the downstream asset body:

@asset(non_argument_deps={AssetKey("some_upstream_asset")})
def downstream_asset(context):
    event_log_entry = context.instance.get_latest_materialization_event(AssetKey("some_upstream_asset"))
    metadata = event_log_entry.dagster_event.event_specific_data.materialization.metadata
    foo_value = metadata["foo"].value
    assert foo_entry.text == "bar"

[dpeng817]

If you want to figure out the latest materialization event for a particular partition key:

@asset(non_argument_deps={AssetKey("some_upstream_asset")})
def downstream_asset(context):
    event_log_record = context.instance.get_event_records(EventRecordsFilter(asset_key=AssetKey("some_upstream_asset"), asset_partitions=["some_partition"]), ascending=False)
    event_log_entry = event_log_record.event_log_entry
    metadata = event_log_entry.dagster_event.event_specific_data.materialization.metadata
    foo_value = metadata["foo"].value
    assert foo_entry.text == "bar"

[sryza]

@sryza I am using assets and I mean context.add_input_metadata

So you want add_input_metadata during load_input and then access it within @asset?

You should be able to do something like

@asset
def downstream_asset(context, upstream_asset):
    latest_observation_record = context.instance.get_event_records(
        EventRecordsFilter(
            event_type=DagsterEventType.ASSET_OBSERVATION,
            asset_key=AssetKey(["upstream_asset"]),
        ),
        ascending=False,
        limit=1,
    )[0]
    metadata = latest_observation_record.event_log_entry.dagster_event.event_specific_data.asset_observation.metadata