SimpleHttpOperator takes lambda to transform response before doing XCom push

Description

I propose that the SimpleHttpOperator also take an argument like xcom_function or xcom_transformation that describes a lambda or single-variable function that takes as input the response to the http request and uses the output of the function for the xcom push.

alternatively, the existing parameter xcom_push can be modified to take in both Booleans and Callables. if provided False or True, behavior is unchanged. If provided a single variable callable, the function is applied to the response.

Use case / motivation

Sometimes we want to XCom push a portion of the response, and then use that portion in a templated xcom pull, where we do not have access to things like json deserialization. Currently the best way to do this is to have a middleman task that takes the response.text xcom from the httpoperator, applies the transformation and then pushes the result.

Are you willing to submit a PR? Yes

Related Issues None that I could find

[potiuk]

Ok. closing this one then.

[Naraujo13]

What do you think about using the response_filter parameter? https://github.com/apache/airflow/blob/8505d2f0a4524313e3eff7a4f16b9a9439c7a79f/airflow/providers/http/operators/http.py#L120-L122

It makes sense, for the HTTPOperator we could simply use the response_filter, so it wouldn’t make sense to add the filter here.

I think I mixed a different use case where one time at work we extended HTTPOperator to save the results on S3, and we wanted to save the data but only push a small part of it (list of ids, etc) to XCom to use in a next request, so we needed different filters there. But this is specific to a different use case.