Could not get scheduler_job_id

Apache Airflow version:

2.0.0

Kubernetes version (if you are using kubernetes) (use kubectl version):

1.18.3

Environment:

Cloud provider or hardware configuration: AWS

What happened:

When trying to run a DAG, it gets scheduled, but task is never run. When attempting to run task manually, it shows an error:

Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.

Python version: 3.8.7
Airflow version: 2.0.0
Node: airflow-web-ffdd89d6-h98vj
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/decorators.py", line 60, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 1366, in run
    executor.start()
  File "/usr/local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 493, in start
    raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id

What you expected to happen:

The task to be run successfully without

How to reproduce it:

Haven’t pinpointed what causes the issue, besides an attempted upgrade from Airflow 1.10.14 to Airflow 2.0.0

Anything else we need to know:

This error is encountered in an upgrade of Airflow from 1.10.14 to Airflow 2.0.0

EDIT: Formatted to fit the issue template

[arijitdey]

I am facing the same error while trying to backfill: Logs:

[2021-01-27 06:35:50,209] {airflow-log-cleanup.py:44} INFO - ENABLE_DELETE_CHILD_LOG  False
/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:320 DeprecationWarning: The statsd_on option in [scheduler] has been moved to the statsd_on option in [metrics] - the old setting has been used, but please update your config.
[2021-01-27 06:35:50,376] {kubernetes_executor.py:491} INFO - Start Kubernetes executor
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 89, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 116, in dag_backfill
    run_backwards=args.run_backwards,
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dag.py", line 1701, in run
    job.run()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 237, in run
    self._execute()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/backfill_job.py", line 788, in _execute
    executor.start()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 493, in start
    raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id

[zhang699]

I also encounter this problem, but curious about another questions, why load same DAG three times before the exception happen, and the last two of loaded DAG path seems incorrect.

[2021-02-19 03:35:50,204] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,205] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..data/code-dag.py
[2021-02-19 03:35:50,222] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,223] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..2021_02_19_03_05_19.306169647/code-dag.py
[2021-02-19 03:35:50,239] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>

airflow@webserver-6b975954d-pf6h8:/opt/airflow$ airflow dags backfill etl_dag1 -s 2020-01-20 -e 2020-01-21
[2021-02-19 03:35:48,536] {settings.py:210} DEBUG - Setting up DB connection pool (PID 214)
[2021-02-19 03:35:48,537] {settings.py:281} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=214
[2021-02-19 03:35:48,636] {cli_action_loggers.py:40} DEBUG - Adding <function default_action_log at 0x7f7b77054320> to pre execution callback
[2021-02-19 03:35:50,016] {cli_action_loggers.py:66} DEBUG - Calling callbacks: [<function default_action_log at 0x7f7b77054320>]
/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py:62 PendingDeprecationWarning: --ignore-first-depends-on-past is deprecated as the value is always set to True
[2021-02-19 03:35:50,050] {dagbag.py:448} INFO - Filling up the DagBag from /home/airflow/dags
[2021-02-19 03:35:50,051] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/code-dag.py
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
[2021-02-19 03:35:50,204] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,205] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..data/code-dag.py
[2021-02-19 03:35:50,222] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,223] {dagbag.py:287} DEBUG - Importing /home/airflow/dags/..2021_02_19_03_05_19.306169647/code-dag.py
[2021-02-19 03:35:50,239] {dagbag.py:413} DEBUG - Loaded DAG <DAG: etl_dag1>
[2021-02-19 03:35:50,240] {executor_loader.py:82} DEBUG - Loading core executor: KubernetesExecutor
[2021-02-19 03:35:50,398] {kubernetes_executor.py:473} INFO - Start Kubernetes executor
[2021-02-19 03:35:50,415] {cli_action_loggers.py:84} DEBUG - Calling callbacks: []
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 89, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 116, in dag_backfill
    run_backwards=args.run_backwards,
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dag.py", line 1706, in run
    job.run()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 237, in run
    self._execute()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/backfill_job.py", line 788, in _execute
    executor.start()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 475, in start
    raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
[2021-02-19 03:35:50,429] {settings.py:292} DEBUG - Disposing DB connection pool (PID 214)