Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Airflow hangs on "Found x duplicates in table task_fail. Will attempt to move them"
See original GitHub issueApache Airflow version
2.3.0 (latest released)
What happened
After upgrade from 2.2.2 to 2.3.0, when I am trying to do airflow db upgrade all I get is following log and then airflow hangs:
Found 16 duplicates in table task_fail. Will attempt to move them.
What you think should happen instead
Airflow should not hang at this log and properly move these duplicates
How to reproduce
- Run Airflow 2.2.2
- Create duplicates in table task_fail
- Update Airflow to 2.3.0 and try upgrading DB
Operating System
Debian 10
Versions of Apache Airflow Providers
apache-airflow-providers-apache-cassandra==2.0.1 \
apache-airflow-providers-apache-hive==2.0.2 \
apache-airflow-providers-celery==2.1.0 \
apache-airflow-providers-cncf-kubernetes==2.0.2 \
apache-airflow-providers-ftp==2.0.1 \
apache-airflow-providers-http==2.0.1 \
apache-airflow-providers-imap==2.0.1 \
apache-airflow-providers-jdbc==2.0.1 \
apache-airflow-providers-mysql==2.1.1 \
apache-airflow-providers-papermill==2.2.3 \
apache-airflow-providers-postgres==2.2.0 \
apache-airflow-providers-sftp==2.1.1 \
apache-airflow-providers-sqlite==2.0.1 \
apache-airflow-providers-ssh==2.1.1 \
apache-airflow-providers-google==5.1.0 \
apache-airflow-providers-apache-beam==3.1.0 \
apache-airflow-providers-apache-spark==2.0.1 \
Deployment
Other Docker-based deployment
Deployment details
Google GKE 2 Instances of Airflow processes running(HA configuration)
Anything else
Problem occurs every time I am trying to update database.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project’s Code of Conduct
Issue Analytics
- State:
- Created a year ago
- Comments:9 (5 by maintainers)
Top Results From Across the Web
Release Notes — Airflow Documentation
New to this release of Airflow is the concept of Datasets to Airflow, and with it a new way of scheduling dags: data-aware...
Read more >Apache Airflow tasks are stuck in a 'up_for_retry' state
I resorted to reading the source code, and found that Airflow treats up_for_retry tasks differently if they are part of a backfill DAG...
Read more >Troubleshooting DAGs | Cloud Composer
The scheduler discovered a duplicate of a task and sends Poison Pill and SIGTERM ... DAG crashes the Airflow web server or causes...
Read more >Azure - Databricks Knowledge Base
Cluster fails to start with dummy does not exist error · Problem You try to start a cluster, but it fails to start....
Read more >Troubleshooting in Athena - AWS Documentation
This error usually occurs when a file is removed when a query is running. Either rerun the query, or check your workflow to...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
https://github.com/apache/airflow/pull/23458 is a life saver! 20 hours db migration becomes 20 seconds! 😂
Sorry for the late reply, I am using PostgreSQL. I assume that I just became impatient and canceled the db upgrade command prematurely. I am not sure why my task_fail table had some rows (85) which did not exist in the task_instance table though