docker-compose up airflow-init produces airflow command error

Apache Airflow version: 2.0.1

Environment:

• OS (e.g. from /etc/os-release): macOS 10.15.7
• Kernel (e.g. uname -a): Darwin Kernel Version 19.6.0
• Install tools: docker-compose

What happened: Following the steps in the Quick start > Running Airflow in Docker page (https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#initializing-environment):

1. Pulled down the pre-made docker-compose.yaml file (https://airflow.apache.org/docs/apache-airflow/2.0.1/docker-compose.yaml)
2. Ran docker-compose up airflow-init within the same directory as the docker-compose file
3. This command failed with: airflow command error: argument GROUP_OR_COMMAND: airflow upgradedb command, has been removed, please use airflow db upgrade, see help above.

What you expected to happen: The latest stable official airflow docker image to be pulled, and airflow-init to complete successfully.

How to reproduce it: For the exact traceback block below, I stopped and removed all of my docker images:

1. docker stop (docker ps -q)
2. docker rm (docker ps -a -q)
3. docker images -a -q | xargs docker rmi -f
4. Follow the instructions per the quick start docs (https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html) to pull down the docker-compose.yaml file, then attempt to run docker-compose up airflow-init.

Anything else we need to know: This is very similar issue to #14143

Below is the traceback block

~/C/P/docker-apache-airflow  docker-compose up airflow-init c̶o̶n̶d̶a̶-̶e̶n̶v̶ Pulling postgres (postgres:13)… 13: Pulling from library/postgres 45b42c59be33: Already exists 40adec129f1a: Pull complete b4c431d00c78: Pull complete 2696974e2815: Pull complete 564b77596399: Pull complete 5044045cf6f2: Pull complete d736e67e6ac3: Pull complete 390c1c9a5ae4: Pull complete c0e62f172284: Pull complete ebcdc659c5bf: Pull complete 29be22cb3acc: Pull complete f63c47038e66: Pull complete 77a0c198cde5: Pull complete c8752d5b785c: Pull complete Digest: sha256:5cfcfb0516ddd317109e2746a462b1dc398d152936640a2b69284629f48dc288 Status: Downloaded newer image for postgres:13 Pulling redis (redis:latest)… latest: Pulling from library/redis 45b42c59be33: Already exists 5ce2e937bf62: Pull complete 2a031498ff58: Pull complete 78f77a50d3fe: Pull complete 60553ab7cb08: Pull complete 68240431d2fc: Pull complete Digest: sha256:f29bcfb891678a0c6a0fc5da0b32ce1ac685af87c0f3aa9327e562da8d3f3b88 Status: Downloaded newer image for redis:latest Pulling airflow-init (apache/airflow:2.0.1)… 2.0.1: Pulling from apache/airflow 45b42c59be33: Already exists f875e16ab19c: Already exists a4fb13de3215: Pull complete f3dd9aaf319e: Pull complete 451a8f7ee895: Pull complete e2d0adf49357: Pull complete 52856765f525: Pull complete fd428150aaa6: Pull complete b4fa7f6b9775: Pull complete b34f0a9234cb: Pull complete 537882d8c1ac: Pull complete dffdc965d748: Pull complete a97e5a67cb46: Pull complete c401f5284516: Pull complete af49f5db1642: Pull complete c61449868181: Pull complete 0b574505c97e: Pull complete a65349ef32b2: Pull complete Digest: sha256:2bf4fe559e9e699487a50d4e587358bb0b3c90b59c8fefeccc8a4fb2e83f3d09 Status: Downloaded newer image for apache/airflow:2.0.1 Creating docker-apache-airflow_postgres_1 … done Creating docker-apache-airflow_redis_1 … done Creating docker-apache-airflow_airflow-init_1 … done Attaching to docker-apache-airflow_airflow-init_1 airflow-init_1 | BACKEND=postgresql+psycopg2 airflow-init_1 | DB_HOST=postgres airflow-init_1 | DB_PORT=5432 airflow-init_1 | airflow-init_1 | DB: postgresql+psycopg2://airflow:***@postgres/airflow airflow-init_1 | [2021-02-22 18:19:41,935] {db.py:674} INFO - Creating tables airflow-init_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl. airflow-init_1 | INFO [alembic.runtime.migration] Will assume transactional DDL. airflow-init_1 | WARNI [airflow.providers_manager] Exception when importing ‘airflow.providers.microsoft.azure.hooks.wasb.WasbHook’ from ‘apache-airflow-providers-microsoft-azure’ package: No module named ‘azure.storage.blob’ airflow-init_1 | WARNI [airflow.providers_manager] Exception when importing ‘airflow.providers.microsoft.azure.hooks.wasb.WasbHook’ from ‘apache-airflow-providers-microsoft-azure’ package: No module named ‘azure.storage.blob’ airflow-init_1 | Traceback (most recent call last): airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/script/base.py”, line 162, in _catch_revision_errors airflow-init_1 | yield airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/script/base.py”, line 356, in upgrade_revs airflow-init_1 | revs = list(revs) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/script/revision.py”, line 904, in iterate_revisions airflow-init_1 | requested_lowers = self.get_revisions(lower) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/script/revision.py”, line 455, in get_revisions airflow-init_1 | return sum([self.get_revisions(id_elem) for id_elem in id], ()) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/script/revision.py”, line 455, in <listcomp> airflow-init_1 | return sum([self.get_revisions(id_elem) for id_elem in id], ()) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/script/revision.py”, line 460, in get_revisions airflow-init_1 | for rev_id in resolved_id airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/script/revision.py”, line 460, in <genexpr> airflow-init_1 | for rev_id in resolved_id airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/script/revision.py”, line 536, in _revision_for_ident airflow-init_1 | resolved_id, airflow-init_1 | alembic.script.revision.ResolutionError: No such revision or branch ‘449b4072c2da’ airflow-init_1 | airflow-init_1 | The above exception was the direct cause of the following exception: airflow-init_1 | airflow-init_1 | Traceback (most recent call last): airflow-init_1 | File “/home/airflow/.local/bin/airflow”, line 8, in <module> airflow-init_1 | sys.exit(main()) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/airflow/main.py”, line 40, in main airflow-init_1 | args.func(args) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py”, line 48, in command airflow-init_1 | return func(*args, **kwargs) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py”, line 89, in wrapper airflow-init_1 | return f(*args, **kwargs) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/db_command.py”, line 48, in upgradedb airflow-init_1 | db.upgradedb() airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/db.py”, line 684, in upgradedb airflow-init_1 | command.upgrade(config, ‘heads’) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/command.py”, line 294, in upgrade airflow-init_1 | script.run_env() airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/script/base.py”, line 481, in run_env airflow-init_1 | util.load_python_file(self.dir, “env.py”) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/util/pyfiles.py”, line 97, in load_python_file airflow-init_1 | module = load_module_py(module_id, path) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/util/compat.py”, line 182, in load_module_py airflow-init_1 | spec.loader.exec_module(module) airflow-init_1 | File “<frozen importlib._bootstrap_external>”, line 678, in exec_module airflow-init_1 | File “<frozen importlib._bootstrap>”, line 219, in _call_with_frames_removed airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/airflow/migrations/env.py”, line 108, in <module> airflow-init_1 | run_migrations_online() airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/airflow/migrations/env.py”, line 102, in run_migrations_online airflow-init_1 | context.run_migrations() airflow-init_1 | File “<string>”, line 8, in run_migrations airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/runtime/environment.py”, line 813, in run_migrations airflow-init_1 | self.get_context().run_migrations(**kw) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/runtime/migration.py”, line 548, in run_migrations airflow-init_1 | for step in self._migrations_fn(heads, self): airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/command.py”, line 283, in upgrade airflow-init_1 | return script.upgrade_revs(revision, rev) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/script/base.py”, line 361, in upgrade_revs airflow-init_1 | for script in reversed(list(revs)) airflow-init_1 | File “/usr/local/lib/python3.6/contextlib.py”, line 99, in exit airflow-init_1 | self.gen.throw(type, value, traceback) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/script/base.py”, line 194, in catch_revision_errors airflow-init_1 | compat.raise(util.CommandError(resolution), from=re) airflow-init_1 | File “/home/airflow/.local/lib/python3.6/site-packages/alembic/util/compat.py”, line 294, in raise airflow-init_1 | raise exception airflow-init_1 | alembic.util.exc.CommandError: Can’t locate revision identified by ‘449b4072c2da’ airflow-init_1 | usage: airflow [-h] GROUP_OR_COMMAND … airflow-init_1 | airflow-init_1 | positional arguments: airflow-init_1 | GROUP_OR_COMMAND airflow-init_1 | airflow-init_1 | Groups: airflow-init_1 | celery Celery components airflow-init_1 | config View configuration airflow-init_1 | connections Manage connections airflow-init_1 | dags Manage DAGs airflow-init_1 | db Database operations airflow-init_1 | kubernetes Tools to help run the KubernetesExecutor airflow-init_1 | pools Manage pools airflow-init_1 | providers Display providers airflow-init_1 | roles Manage roles airflow-init_1 | tasks Manage tasks airflow-init_1 | users Manage users airflow-init_1 | variables Manage variables airflow-init_1 | airflow-init_1 | Commands: airflow-init_1 | cheat-sheet Display cheat sheet airflow-init_1 | info Show information about current Airflow and environment airflow-init_1 | kerberos Start a kerberos ticket renewer airflow-init_1 | plugins Dump information about loaded plugins airflow-init_1 | rotate-fernet-key airflow-init_1 | Rotate encrypted connection credentials and variables airflow-init_1 | scheduler Start a scheduler instance airflow-init_1 | sync-perm Update permissions for existing roles and DAGs airflow-init_1 | version Show the version airflow-init_1 | webserver Start a Airflow webserver instance airflow-init_1 | airflow-init_1 | optional arguments: airflow-init_1 | -h, --help show this help message and exit airflow-init_1 | airflow-init_1 | airflow command error: argument GROUP_OR_COMMAND: airflow upgradedb command, has been removed, please use airflow db upgrade, see help above. airflow-init_1 | [2021-02-22 18:19:50,692] {providers_manager.py:299} WARNING - Exception when importing ‘airflow.providers.microsoft.azure.hooks.wasb.WasbHook’ from ‘apache-airflow-providers-microsoft-azure’ package: No module named ‘azure.storage.blob’ airflow-init_1 | [2021-02-22 18:19:51,231] {providers_manager.py:299} WARNING - Exception when importing ‘airflow.providers.microsoft.azure.hooks.wasb.WasbHook’ from ‘apache-airflow-providers-microsoft-azure’ package: No module named ‘azure.storage.blob’ airflow-init_1 | [2021-02-22 18:19:52,417] {providers_manager.py:299} WARNING - Exception when importing ‘airflow.providers.microsoft.azure.hooks.wasb.WasbHook’ from ‘apache-airflow-providers-microsoft-azure’ package: No module named ‘azure.storage.blob’ airflow-init_1 | [2021-02-22 18:19:52,514] {providers_manager.py:299} WARNING - Exception when importing ‘airflow.providers.microsoft.azure.hooks.wasb.WasbHook’ from ‘apache-airflow-providers-microsoft-azure’ package: No module named ‘azure.storage.blob’ airflow-init_1 | airflow already exist in the db airflow-init_1 | 2.0.1 docker-apache-airflow_airflow-init_1 exited with code 0

[chrisrfu]

It worked! I increased the memory from 2GB to 4GB and kept CPU at 8, Swap at 1GB, and Disk image size at 59.6GB (my default Docker settings) Thank you so much for your help!

[kaxil]

You can ignore the following line:

airflow-webserver_1 | [2021-02-24 16:42:01,329] {providers_manager.py:299} WARNING - Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'

The container exited because it did not have enough memory:

docker-apache-airflow-201_airflow-webserver_1 exited with code 137

Try increasing the memory: https://docs.docker.com/docker-for-mac/space/