Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Support for set in XCom serialization

See original GitHub issue

Description

Add support for set in Xcom serialization method. This requires support for set in nested structures.

Use case / motivation Once we switched to JSON as main serialization format the following error appeared:

TypeError: Object of type 'set' is not JSON serializable

Issue Analytics

State:
Created 3 years ago
Comments:14 (14 by maintainers)

Top GitHub Comments

2reactions

kaxilcommented, Jul 2, 2021

https://github.com/apache/airflow/pull/16786 should fix this

1reaction

kaxilcommented, Sep 14, 2021

Let’s use Custom XCom backend if needed – this isn’t critical

Top Results From Across the Web

XComs — Airflow Documentation

An XCom is identified by a key (essentially its name), as well as the task_id and dag_id it came from. They can have...

Airflow XCOM : The Ultimate Guide - Marc Lamberti

The Airflow XCom is not an easy concept, so let me illustrate why it might be ... Keep in mind that your value...

How to solve this Problem Could not serialize the XCom value ...

so xcom doesn't support exchanging non serialized data type like dataframes (read more about Python pickle here), but if you must , you...

Custom XCom backends | Astronomer Documentation

The final step to setting up your custom XCom backend is configuring your serialization and deserialization methods and applying them to your Airflow ......

Customizing Xcom to enhance data sharing between tasks

Presented by Vikram Koka & Ephraim Anierobi at Airflow Summit 2021.In Apache Airflow, Xcom is the default mechanism for passing data between ...