macos python3.8 AttributeError: Can't pickle local object 'CeleryExecutor.trigger_tasks.<locals>.reset_signals'

Apache Airflow version: 1.10.14

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

System Version: macOS 11.1 (20C69) Version: Darwin 20.2.0 Python 3.8.5

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened: [2021-02-04 13:55:00,605] {base_executor.py:58} INFO - Adding to queue: [‘airflow’, ‘run’, ‘example_bash_operator’, ‘runme_2’, ‘2021-02-04T05:54:56.206452+00:00’, ‘–local’, ‘–pool’, ‘default_pool’, ‘-sd’, ‘/Users/saith/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/example_dags/example_bash_operator.py’] [2021-02-04 13:55:00,636] {scheduler_job.py:1384} ERROR - Exception when executing execute_helper Traceback (most recent call last): File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py”, line 1382, in _execute self._execute_helper() File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py”, line 1453, in _execute_helper if not self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag): File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py”, line 1515, in _validate_and_run_task_instances self.executor.heartbeat() File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/executors/base_executor.py”, line 130, in heartbeat self.trigger_tasks(open_slots) File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/executors/celery_executor.py”, line 229, in trigger_tasks send_pool = Pool(processes=num_processes, initializer=reset_signals) File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/multiprocessing/context.py”, line 119, in Pool return Pool(processes, initializer, initargs, maxtasksperchild, File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/multiprocessing/pool.py”, line 212, in init self._repopulate_pool() File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/multiprocessing/pool.py”, line 303, in _repopulate_pool return self._repopulate_pool_static(self._ctx, self.Process, File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/multiprocessing/pool.py”, line 326, in _repopulate_pool_static w.start() File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/multiprocessing/process.py”, line 121, in start self._popen = self._Popen(self) File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/multiprocessing/context.py”, line 284, in _Popen return Popen(process_obj) File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/multiprocessing/popen_spawn_posix.py”, line 32, in init super().init(process_obj) File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/multiprocessing/popen_fork.py”, line 19, in init self._launch(process_obj) File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/multiprocessing/popen_spawn_posix.py”, line 47, in _launch reduction.dump(process_obj, fp) File “/Users/saith/anaconda3/envs/airflow/lib/python3.8/multiprocessing/reduction.py”, line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can’t pickle local object ‘CeleryExecutor.trigger_tasks.<locals>.reset_signals’

What you expected to happen: the demo dag should be triggered with no error

I think the error is similar to this issue https://issues.apache.org/jira/browse/AIRFLOW-6529

How to reproduce it:

airflow install

conda create -n airflow python=3.8
pip install 'apache-airflow==1.10.14'
pip install 'apache-airflow[redis]==1.10.14'
pip install 'apache-airflow[celery]==1.10.14'
pip install 'apache-airflow[mysql]==1.10.14'

change the airflow.cfg

sql_alchemy_conn = mysql://airflow:airflow@localhost:3306/airflow
broker_url = redis://127.0.0.1:6379/0
result_backend = db+mysql://airflow:airflow@localhost:3306/airflow
executor = CeleryExecutor

boot mysql， redis

mysql.server start
redis-server

CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;

airflow initdb

airflow webserver
airflow worker
airflow scheduler

when i triggered on demo dag , the error occurs

Anything else we need to know:

the same environment , i use conda python 3.7.9 the error is disappear

[leonsmith]

Ditto same issue, did a bit of digging below.

So this looks to be a change in 3.8 on macosx and the default for multiprocessing changing.

We have AIRFLOW__CORE__MP_START_METHOD to tweak this behaviour in the scheduler but not such control in the celery executor.

You can however bypass this by running with AIRFLOW__CELERY__SYNC_PARALLELISM=1

[sighingnow]

I got the same problem too, is it an issue? My machine: MacOs 11.3, python 3.8.7, mysql 8, airflow 2.1.0

It is no longer a problem on master.