"duplicate key value violates unique constraint "dag_run_dag_id_execution_date_key" when triggering a DAG

Apache Airflow version: 2.0.0

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.14

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

[2021-04-02 07:23:30,513] [ERROR] app.py:1892 - Exception on /api/v1/dags/auto_test/dagRuns [POST] Traceback (most recent call last): File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1277, in _execute_context cursor, statement, parameters, context File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py”, line 608, in do_execute cursor.execute(statement, parameters) psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint “dag_run_dag_id_execution_date_key” DETAIL: Key (dag_id, execution_date)=(auto_test, 1967-12-13 20:57:42.043+01) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “/usr/local/lib/python3.6/site-packages/flask/app.py”, line 2447, in wsgi_app response = self.full_dispatch_request() File “/usr/local/lib/python3.6/site-packages/flask/app.py”, line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File “/usr/local/lib/python3.6/site-packages/flask/app.py”, line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File “/usr/local/lib/python3.6/site-packages/flask/_compat.py”, line 39, in reraise raise value File “/usr/local/lib/python3.6/site-packages/flask/app.py”, line 1950, in full_dispatch_request rv = self.dispatch_request() File “/usr/local/lib/python3.6/site-packages/flask/app.py”, line 1936, in dispatch_request return self.view_functionsrule.endpoint File “/usr/local/lib/python3.6/site-packages/connexion/decorators/decorator.py”, line 48, in wrapper response = function(request) File “/usr/local/lib/python3.6/site-packages/connexion/decorators/uri_parsing.py”, line 144, in wrapper response = function(request) File “/usr/local/lib/python3.6/site-packages/connexion/decorators/validation.py”, line 184, in wrapper response = function(request) File “/usr/local/lib/python3.6/site-packages/connexion/decorators/validation.py”, line 384, in wrapper return function(request) File “/usr/local/lib/python3.6/site-packages/connexion/decorators/response.py”, line 103, in wrapper response = function(request) File “/usr/local/lib/python3.6/site-packages/connexion/decorators/parameter.py”, line 121, in wrapper return function(**kwargs) File “/usr/local/lib/python3.6/site-packages/airflow/api_connexion/security.py”, line 47, in decorated return func(*args, **kwargs) File “/usr/local/lib/python3.6/site-packages/airflow/utils/session.py”, line 65, in wrapper return func(*args, session=session, **kwargs) File “/usr/local/lib/python3.6/site-packages/airflow/api_connexion/endpoints/dag_run_endpoint.py”, line 231, in post_dag_run session.commit() File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 1046, in commit self.transaction.commit() File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 504, in commit self._prepare_impl() File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 483, in _prepare_impl self.session.flush() File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 2540, in flush self._flush(objects) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 2682, in _flush transaction.rollback(capture_exception=True) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py”, line 70, in exit with_traceback=exc_tb, File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 182, in raise raise exception File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 2642, in _flush flush_context.execute() File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py”, line 422, in execute rec.execute(self) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py”, line 589, in execute uow, File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py”, line 245, in save_obj insert, File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py”, line 1136, in _emit_insert_statements statement, params File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1011, in execute return meth(self, multiparams, params) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py”, line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1130, in _execute_clauseelement distilled_params, File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1317, in execute_context e, statement, parameters, cursor, context File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1511, in handle_dbapi_exception sqlalchemy_exception, with_traceback=exc_info[2], from=e File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 182, in raise raise exception File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1277, in _execute_context cursor, statement, parameters, context File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py”, line 608, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint “dag_run_dag_id_execution_date_key” DETAIL: Key (dag_id, execution_date)=(auto_test, 1967-12-13 20:57:42.043+01) already exists.

[SQL: INSERT INTO dag_run (dag_id, execution_date, start_date, end_date, state, run_id, creating_job_id, external_trigger, run_type, conf, last_scheduling_decision, dag_hash) VALUES (%(dag_id)s, %(execution_date)s, %(start_date)s, %(end_date)s, %(state)s, %(run_id)s, %(creating_job_id)s, %(external_trigger)s, %(run_type)s, %(conf)s, %(last_scheduling_decision)s, %(dag_hash)s) RETURNING dag_run.id] [parameters: {‘dag_id’: ‘auto_test’, ‘execution_date’: datetime.datetime(1967, 12, 13, 19, 57, 42, 43000, tzinfo=Timezone(‘UTC’)), ‘start_date’: datetime.datetime(2021, 4, 2, 7, 23, 30, 511735, tzinfo=Timezone(‘UTC’)), ‘end_date’: None, ‘state’: ‘running’, ‘run_id’: ‘dag_run_id_zstp_4435_postman11’, ‘creating_job_id’: None, ‘external_trigger’: True, ‘run_type’: <DagRunType.MANUAL: ‘manual’>, ‘conf’: <psycopg2.extensions.Binary object at 0x7f07b30b71e8>, ‘last_scheduling_decision’: None, ‘dag_hash’: None}]

What you expected to happen: The second trigger success

How to reproduce it: Trigger a dag with the following conf:

{ "dag_run_id": "dag_run_id_zstp_4435_postman", "execution_date": "1967-12-13T19:57:42.043Z", "conf": {} }

Then trigerring the same dag only with a different dag_run_id:

{ "dag_run_id": "dag_run_id_zstp_4435_postman111", "execution_date": "1967-12-13T19:57:42.043Z", "conf": {} }

Anything else we need to know:

[potiuk]

Seems strange to have that unique constraint. If two DAG runs have different run_ids, why not let them run at the same time?

Because historically execution date was the key not run_id and the whole datamodel was based on execution_date being part of foreign key in multiple tables. This has since been removed and in Airflow 2.2 the primary key is run_id and execution_date is just a field. See the changelog https://airflow.apache.org/docs/apache-airflow/2.2.4/changelog.html#airflow-2-2-0-2021-10-11

[vikramcse]

Hi @kaxil and @ephraimbuddy

I am able to reproduce the issue and tried to resolve it.

Below is the error message returned by the /dagRuns api. My question is does the below returned message detail text is correct? please let me know if it needs some change.

{
    "detail": "DAGRun with DAG ID: 'taskflow' and DAGRun ExecutionDate '2021-04-03 07:50:16.317733+00:00' already exists",
    "status": 409,
    "title": "Conflict",
    "type": "http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/stable-rest-api-ref.html#section/Errors/AlreadyExists"
}

Thank You