Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Clearing Tasks throws TypeError: can't pickle thread.lock objects

See original GitHub issue

There’s an issue we’re seeing fairly frequently when attempting to clear a task using the UI. It doesn’t happen 100% of the time, and sometimes killing and restarting all airflow processes eliminates the issue. If this happens, it’s pretty much impossible to clear the task state without a hard reboot at least. Here is the stack trace we’re seeing:

Traceback (most recent call last):
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/flask_admin/base.py", line 68, in inner
    return self._run_view(f, *args, **kwargs)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/flask_admin/base.py", line 359, in _run_view
    return fn(self, *args, **kwargs)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/flask_login.py", line 755, in decorated_view
    return func(*args, **kwargs)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/airflow/www/utils.py", line 103, in wrapper
    return f(*args, **kwargs)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/airflow/www/utils.py", line 152, in wrapper
    return f(*args, **kwargs)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/airflow/www/views.py", line 901, in clear
    include_upstream=upstream)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/airflow/models.py", line 2345, in sub_dag
    dag = copy.deepcopy(self)
  File "/home/y/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/airflow/models.py", line 2331, in __deepcopy__
    setattr(result, k, copy.deepcopy(v, memo))
  File "/home/y/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/home/y/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/home/y/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/airflow/models.py", line 1607, in __deepcopy__
    setattr(result, k, copy.deepcopy(v, memo))
  File "/home/y/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/home/y/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/home/y/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/airflow/models.py", line 1607, in __deepcopy__
    setattr(result, k, copy.deepcopy(v, memo))
  File "/home/y/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/home/y/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/home/y/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/home/y/share/airflow_venv/lib/python2.7/site-packages/airflow/models.py", line 1607, in __deepcopy__
    setattr(result, k, copy.deepcopy(v, memo))
  File "/home/y/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/home/y/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/y/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/home/y/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/y/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/home/y/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/y/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/home/y/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/y/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/home/y/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/y/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/home/y/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/y/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/home/y/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/home/y/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/home/y/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/y/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/home/y/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/y/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/home/y/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/y/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/home/y/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/y/lib/python2.7/copy.py", line 182, in deepcopy
    rv = reductor(2)
TypeError: can't pickle thread.lock objects

Issue Analytics

State:
Created 8 years ago
Comments:11 (7 by maintainers)

Top GitHub Comments

1reaction

adambomcommented, Mar 21, 2016

This has something to do with subdags. When I removed the subdag this issue disappeared. The two dags had different schedule intervals (hourly and daily), which may have had something to do with it.

I’ll leave it open for now in case someone else runs into the same problem, but for the time being I got around this by splitting the subdag out into its own dag.

0reactions

kaxilcommented, May 13, 2021

@axelet @david30907d Can one of you create a separate Github issue for it please

Top Results From Across the Web

Airflow (Google Composer) TypeError: can't pickle _thread ...

What I have tried? From airflow UI interface to clear task, the task, not work; From command like use command backfill, not work;...

Clearing task in UI causes TypeError: can't pickle thread.lock ...

I resolved the problem by going to 'Browse' -> 'Task Instances' -> check the instances that I wanted to clear -> With selected...

cannot pickle '_thread.lock' object - You.com | The Search ...

The problem here is that self in function run_parallel() can't be pickled as it is a class instance. Moving this parallelized function run_parallel()...

apache/incubator-airflow - Gitter

Clearing Tasks throws TypeError : can't pickle thread.lock objects. Using airflow 1.8.2 and celery executor. Any ideas on what might be happening?

2052037 – TypeError: can't pickle _thread.RLock objects ...

TypeError : can't pickle _thread.RLock objects Sadly, Python isn't helpful here to explain the reason of the serialization failure. There can be ...