SerializedDagNotFound - when create 1000 dags

Apache Airflow version

2.1.3

Operating System

Linux

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

Community Helm chart 8.0.8

What happened

I have created 1000 dags like that

test_AIR10001.py … test_AIR11001.py

After wait period of 5 minutes try to open any dag and can’t. Here is an exception details: https://sentry.io/share/issue/b97fca62ccec47709e814e70deb3e4ba/

If I try do do 5 dags like that I do not have this issue.

What you expected to happen

I expect dags would be loaded to Ariflow without any issue

How to reproduce

dag code

import numpy as np
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
from src.helpers import dag_helper

with DAG(
    dag_id=dag_helper.get_dag_id_from_path(__file__),
    schedule_interval=None,
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=["example"]
) as dag:
    def print_array():
        """Print Numpy array."""
        a = np.arange(15).reshape(3, 5)
        print(a)
        time_wait_cuy = dag_helper.get_time_wait_cuy()
        print(time_wait_cuy)
        return time_wait_cuy


    run_this = PythonOperator(
        task_id="print_the_context",
        python_callable=print_array,
    )

Here is dag_helper

import os
from pathlib import Path


def get_time_wait_cuy():
    return os.environ['environment']


def get_dag_id_from_path(dag_file_path):
    path = Path(dag_file_path)
    stem = path.stem
    trimmed_path = Path(*path.parts[5:len(path.parts) - 1])
    prefix = str(trimmed_path).replace(os.path.sep, '_')
    return '__'.join([prefix, stem])

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project’s Code of Conduct

[denysivanov]

@khalidmammadov I did some tests on my side and it is environmental issue on my end.

Thanks for support.

[potiuk]

Well, I think It’s a long document.

Coudl you please come up with the digest of it and what is you do not expec there? What is your expectation and problems you see?