Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Dynamic Task Mapping not working with op_kwargs in PythonOperator

See original GitHub issue

Apache Airflow version

2.3.0 (latest released)

What happened

The following DAG was written and expected to generate 3 tasks (one for each string in the list)

dag_code

import logging
from airflow.decorators import dag, task
from airflow.operators.python import PythonOperator

from airflow.utils.dates import datetime


def log_strings_operator(string, *args, **kwargs):
    logging.info("we've made it into the method")
    logging.info(f"operator log - {string}")


@dag(
    dag_id='dynamic_dag_test',
    schedule_interval=None,
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=['example', 'dynamic_tasks']
)
def tutorial_taskflow_api_etl():
    op2 = (PythonOperator
           .partial(task_id="logging_with_operator_task",
                    python_callable=log_strings_operator)
           .expand(op_kwargs=[{"string": "a"}, {"string": "b"}, {"string": "c"}]))

    return op2


tutorial_etl_dag = tutorial_taskflow_api_etl()

error message

Broken DAG: [/usr/local/airflow/dags/dynamic_dag_test.py] Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 343, in _serialize
    return SerializedBaseOperator.serialize_mapped_operator(var)
  File "/usr/local/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 608, in serialize_mapped_operator
    assert op_kwargs[Encoding.TYPE] == DAT.DICT
TypeError: list indices must be integers or slices, not Encoding

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 1105, in to_dict
    json_dict = {"__version": cls.SERIALIZER_VERSION, "dag": cls.serialize_dag(var)}
  File "/usr/local/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 1013, in serialize_dag
    raise SerializationError(f'Failed to serialize DAG {dag.dag_id!r}: {e}')
airflow.exceptions.SerializationError: Failed to serialize DAG 'dynamic_dag_test': list indices must be integers or slices, not Encoding