Occasional "KeyError" in dag_processing

Apache Airflow version: 2.0.0rc2 Environment: Breeze with example dags, Python 3.8 postgres.

• OS (e.g. from /etc/os-release): Linux
• Kernel (e.g. uname -a): Breeze CI image
• Install tools: Breeze:
• Executor: LocalExecutor

./breeze start-airflow --backend postgres --load-example-dags --load-default-connections --install-airflow-version 2.0.0rc2 --skip-mounting-local-sources --python 3.8

What happened:

When testing airflow logging I occasionally stumble upon “KeyError’ from dag_procesing.py. I am not sure exactly when it happens. It’s not always reproducible but it looks like it is when I restart scheduler and trigger 'example_bash_operator.py” it happens rather randomly (1/10 times more or less). It does not happen always when I triggere task manually. DAG gets correctly executed after triggering, but the log is there and warniing printed in the logs right after the DAG finishes execution.

The error I see in scheduler’s logs:

[2020-12-13 19:35:33,752] {dagbag.py:440} INFO - Filling up the DagBag from /usr/local/lib/python3.8/site-packages/airflow/example_dags/example_bash_operator.py                                                                                                                                          
Running <TaskInstance: example_bash_operator.run_after_loop 2020-12-13T19:35:30.648020+00:00 [queued]> on host 6611da4b1a27
[2020-12-13 19:35:34,517] {dagrun.py:444} INFO - Marking run <DagRun example_bash_operator @ 2020-12-13 19:35:30.648020+00:00: manual__2020-12-13T19:35:30.648020+00:00, externally triggered: True> successful                                                                                           
[2020-12-13 19:35:34,523] {scheduler_job.py:1193} INFO - Executor reports execution of example_bash_operator.run_after_loop execution_date=2020-12-13 19:35:30.648020+00:00 exited with status success for try_number 1                                                                                   
Process ForkProcess-34:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 365, in _run_processor_manager
    processor_manager.start()
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 596, in start
    return self._run_parsing_loop()
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 659, in _run_parsing_loop
    self._processors.pop(processor.file_path)
KeyError: '/usr/local/lib/python3.8/site-packages/airflow/example_dags/example_bash_operator.py'
[2020-12-13 19:35:35,589] {dag_processing.py:396} WARNING - DagFileProcessorManager (PID=1029759) exited with exit code 1 - re-launching

What you expected to happen:

No error in logs.

How to reproduce it:

./breeze start-airflow --backend postgres --load-example-dags --load-default-connections --install-airflow-version 2.0.0rc2 --skip-mounting-local-sources --python 3.8

Login to the webserver, enable 'example_bash_operator", wait for it to execute. Trigger the example DAG several times (always wait for the end of execution. It happens randomly (for me around 1/10 tasks)

Anything else we need to know:

[kaxil]

Yeah I have seen this error - will need to dig in a bit – but is a non-harmful one.

Agree. Until we start having an influx of questions about all the warnings 😃

Indeed, I will take a look at it soon’ish – and hopefully we can merge a fix so we can tell them the fix is merged – 2.0.1 will contain the fix

[kaxil]

Hi Team ,

Our dags are failing to start with the below issue . We have not done any chages , had just restarted our scheduler and then onwards we are facing this issue.

[2021-03-01 15:27:34,622] {scheduler_job.py:1229} ERROR - Executor reports task instance <TaskInstance: wmt.hw.clinic.create_cluster.create_cluster 2021-03-01 15:06:37.473524+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
[2021-03-01 15:27:36,366] {dagrun.py:429} ERROR - Marking run <DagRun wmt.hw.clinic.create_cluster @ 2021-03-01 15:06:37.473524+00:00: manual__2021-03-01T15:06:37.473524+00:00, externally triggered: True> failed
Process ForkProcess-8:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
  self.run()
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
  self._target(*self._args, **self._kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 365, in _run_processor_manager
  processor_manager.start()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 596, in start
  return self._run_parsing_loop()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 659, in _run_parsing_loop
  self._processors.pop(processor.file_path)
KeyError: '/opt/airflow/dags/a0s03fc/rcm_clinic_create_cluster.py'

You need to update to 2.0.1