Kubernetes Objects are not serializable and break Graph View in UI

Apache Airflow version: 2.0.1

Kubernetes version (if you are using kubernetes) (use kubectl version): v1.17.12

Environment:

• Cloud provider or hardware configuration: AWS EKS
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened: When you click on Graph view for some DAGs in 2.0.1 the UI errors out (logs below).

What you expected to happen: The Graph view to display

How to reproduce it: It is not clear to me. This only happening on a handle of our DAGs. Also the tree view displays fine.

Anything else we need to know:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/decorators.py", line 97, in view_func
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/decorators.py", line 60, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/session.py", line 65, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 2080, in graph
    return self.render_template(
  File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 396, in render_template
    return super().render_template(
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/baseviews.py", line 280, in render_template
    return render_template(
  File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 137, in render_template
    return _render(
  File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 120, in _render
    rv = template.render(context)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/templates/airflow/graph.html", line 21, in top-level template code
    {% from 'appbuilder/loading_dots.html' import loading_dots %}
  File "/usr/local/lib/python3.8/site-packages/airflow/www/templates/airflow/dag.html", line 21, in top-level template code
    {% from 'appbuilder/dag_docs.html' import dag_docs %}
  File "/usr/local/lib/python3.8/site-packages/airflow/www/templates/airflow/main.html", line 20, in top-level template code
    {% extends 'appbuilder/baselayout.html' %}
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
    {% import 'appbuilder/baselib.html' as baselib %}
  File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 60, in top-level template code
    {% block tail %}
  File "/usr/local/lib/python3.8/site-packages/airflow/www/templates/airflow/graph.html", line 145, in block "tail"
    var task_instances = {{ task_instances|tojson }};
  File "/usr/local/lib/python3.8/site-packages/flask/json/__init__.py", line 376, in tojson_filter
    return Markup(htmlsafe_dumps(obj, **kwargs))
  File "/usr/local/lib/python3.8/site-packages/flask/json/__init__.py", line 290, in htmlsafe_dumps
    dumps(obj, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask/json/__init__.py", line 211, in dumps
    rv = _json.dumps(obj, **kwargs)
  File "/usr/local/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/json.py", line 74, in _default
    raise TypeError(f"Object of type '{obj.__class__.__name__}' is not JSON serializable")
TypeError: Object of type 'V1ResourceRequirements' is not JSON serializable

[Mokubyow]

Just wanted to chime in here since I had the same exact issue. I found the issue was only present when using the deprecated executor_config format. When I upgraded to using the pod_override with the k8s models my graph view was able to render. So it appears to be due to something in the backward compatibility process. Example:

Broken…

PythonOperator(
    task_id=f"sync_{table_name}",
    python_callable=sync_table,
    provide_context=True,
    op_kwargs={"table_name": table_name},
    executor_config={"KubernetesExecutor": {"request_cpu": "1"}},
    retries=5,
    dag=dag,
)

Works!

PythonOperator(
    task_id=f"sync_{table_name}",
    python_callable=sync_table,
    provide_context=True,
    op_kwargs={"table_name": table_name},
    executor_config={
        "pod_override": k8s.V1Pod(
            spec=k8s.V1PodSpec(
                containers=[
                    k8s.V1Container(
                        name="base",
                        resources=k8s.V1ResourceRequirements(requests={"cpu": "1"}),
                    )
                ]
            )
        )
    },
    retries=5,
    dag=dag,
)

[uranusjr]

task_instances is built from alchemy_to_dict(), which does not promise to return a JSON-serialisable object. So some additional processing is needed. (I don’t think it’s a good idea to make the change directly in alchemy_to_dict() either since the function is used for things not required to be JSON-serialisable.)