Airflow webserver does not work with Python >=3.5

When starting the Airflow web server at http://localhost:8080/admin/ (inside a virtual environment with Python 3), the following error message occurs:

Traceback (most recent call last):
  File "/Users/ian/taxi_pipeline/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
    cursor, statement, parameters, context
  File "/Users/ian/taxi_pipeline/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such column: dag.description

The problem is here:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column: dag.description [SQL: 'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners, dag.description AS dag_description, dag.default_view AS dag_default_view, dag.schedule_interval AS dag_schedule_interval \nFROM dag \nWHERE dag.is_subdag = 0 AND dag.is_active = 1 ORDER BY dag.dag_id\n LIMIT ? OFFSET ?'] [parameters: (100, 0)] (Background on this error at: http://sqlalche.me/e/e3q8)

To reproduce, I ran the following commands according to the instructions:

cd ~
rm -rf taxi && rm -rf taxi_pipeline && rm -rf airflow && rm -rf tfx
git clone https://github.com/tensorflow/tfx.git
cd tfx && git checkout tags/0.13.0 -b release

cd ~
virtualenv -p python3.5 taxi_pipeline
source ./taxi_pipeline/bin/activate

export AIRFLOW_HOME=~/airflow
export TAXI_DIR=~/taxi
export TFX_DIR=~/tfx

pip install tensorflow==1.13.1
pip install docker
export SLUGIFY_USES_TEXT_UNIDECODE=yes
pip install apache-airflow
pip install tfx==0.13.0

airflow initdb

cd $TFX_DIR/tfx/examples/chicago_taxi_pipeline

mkdir -p $TAXI_DIR/data/simple
cp data/simple/data.csv $TAXI_DIR/data/simple

mkdir -p $AIRFLOW_HOME/dags/taxi
cp taxi_pipeline_simple.py $AIRFLOW_HOME/dags/taxi

cp taxi_utils.py $TAXI_DIR

airflow webserver

I see the same behaviour for a Python 3.7 virtual environment. Oddly, enough I get the same error now for 2.7!

I assumed it was a version incompatibility with Airflow, so I fixed the version to 1.10.2, but that yields:

Traceback (most recent call last):
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_admin/base.py", line 69, in inner
    return self._run_view(f, *args, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_admin/base.py", line 368, in _run_view
    return fn(self, *args, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_login/utils.py", line 258, in decorated_view
    return func(*args, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/utils/db.py", line 73, in wrapper
    return func(*args, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/www/views.py", line 2224, in index
    auto_complete_data=auto_complete_data)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_admin/base.py", line 308, in render
    return render_template(template, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/templating.py", line 134, in render_template
    context, ctx.app)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask/templating.py", line 116, in _render
    rv = template.render(context)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/www/templates/airflow/dags.html", line 18, in top-level template code
    {% extends "airflow/master.html" %}
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/www/templates/airflow/master.html", line 18, in top-level template code
    {% extends "admin/master.html" %}
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/www/templates/admin/master.html", line 18, in top-level template code
    {% extends 'admin/base.html' %}
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_admin/templates/bootstrap3/admin/base.html", line 81, in top-level template code
    {% block tail_js %}
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/airflow/www/templates/admin/master.html", line 44, in block "tail_js"
    xhr.setRequestHeader("X-CSRFToken", "{{ csrf_token() }}");
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/flask_wtf/csrf.py", line 47, in generate_csrf
    setattr(g, field_name, s.dumps(session[field_name]))
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/itsdangerous/serializer.py", line 166, in dumps
    payload = want_bytes(self.dump_payload(obj))
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/itsdangerous/url_safe.py", line 42, in dump_payload
    json = super(URLSafeSerializerMixin, self).dump_payload(obj)
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/itsdangerous/serializer.py", line 133, in dump_payload
    return want_bytes(self.serializer.dumps(obj, **self.serializer_kwargs))
  File "/Users/ian/taxi_pipeline/lib/python3.7/site-packages/itsdangerous/_json.py", line 18, in dumps
    return json.dumps(obj, **kwargs)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

However, with the version fixed it seems to work with Python 3.5, just not 3.6 or 3.7. Since internally we use 3.6+ this means we’d have to downgrade (not recommended) or stick with Python 2.7, which is not ideal as support runs out in 2020.

Is this expected after support for Python 3(.5) was added?

[zhitaoli]

Just a note in case anyone ran into this issue: if you see an error related to “TypeError: Object of type bytes is not JSON serializable” and you have ever used more than one python versions w/ airflow, clearing cookie or open the airflow webserver in an incognito page of Chrome may fix the problem.

[ianhellstrom]

I have not checked it and since it’s not officially supported anyway (due to Beam) I do not see an immediate. We can probably close this issue.