Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
bug: Airflow 1.10.x and 2.1.x broken when upgrading to Meltano 2.0+
See original GitHub issueI’m having issues deploying to production my 2.0.2 version of meltano regarding airflow.
My current version of airflow is 1.10.15 (not 2.X) and everything works locally, but when I push to production I get this error message:
15/06/2022, 11:32:18 [2m2022-06-15T15:32:18.287237Z[0m [[32m[1mdebug [0m] [1mDeleted configuration at /project/.meltano/run/airflow/airflow.cfg[0m airflow-scheduler
15/06/2022, 11:32:18 [2m2022-06-15T15:32:18.287889Z[0m [[31m[1merror [0m] [1mTraceback (most recent call last): airflow-scheduler
15/06/2022, 11:32:18 File "/project/.meltano/orchestrators/airflow/venv/bin/airflow", line 25, in <module> airflow-scheduler
15/06/2022, 11:32:18 from airflow.configuration import conf airflow-scheduler
15/06/2022, 11:32:18 File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/__init__.py", line 31, in <module> airflow-scheduler
15/06/2022, 11:32:18 from airflow.utils.log.logging_mixin import LoggingMixin airflow-scheduler
15/06/2022, 11:32:18 File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/utils/__init__.py", line 24, in <module> airflow-scheduler
15/06/2022, 11:32:18 from .decorators import apply_defaults as _apply_defaults airflow-scheduler
15/06/2022, 11:32:18 File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/utils/decorators.py", line 36, in <module> airflow-scheduler
15/06/2022, 11:32:18 from airflow import settings airflow-scheduler
15/06/2022, 11:32:18 File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/settings.py", line 38, in <module> airflow-scheduler
15/06/2022, 11:32:18 from airflow.configuration import conf, AIRFLOW_HOME, WEBSERVER_CONFIG # NOQA F401 airflow-scheduler
15/06/2022, 11:32:18 File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 737, in <module> airflow-scheduler
15/06/2022, 11:32:18 conf.read(AIRFLOW_CONFIG) airflow-scheduler
15/06/2022, 11:32:18 File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 425, in read airflow-scheduler
15/06/2022, 11:32:18 self._validate() airflow-scheduler
15/06/2022, 11:32:18 File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 217, in _validate airflow-scheduler
15/06/2022, 11:32:18 self._validate_config_dependencies() airflow-scheduler
15/06/2022, 11:32:18 File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 249, in _validate_config_dependencies airflow-scheduler
15/06/2022, 11:32:18 raise AirflowConfigException( airflow-scheduler
15/06/2022, 11:32:18 airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor airflow-scheduler
15/06/2022, 11:32:18 [0m airflow-scheduler
15/06/2022, 11:32:18 [2m2022-06-15T15:32:18.289815Z[0m [[32m[1mdebug [0m] [1mCommand `airflow --help` failed[0m airflow-scheduler
15/06/2022, 11:32:18 Traceback (most recent call last): airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/cli/__init__.py", line 55, in _run_cli airflow-scheduler
15/06/2022, 11:32:18 cli(obj={"project": None}) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__ airflow-scheduler
15/06/2022, 11:32:18 return self.main(*args, **kwargs) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main airflow-scheduler
15/06/2022, 11:32:18 rv = self.invoke(ctx) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke airflow-scheduler
15/06/2022, 11:32:18 return _process_result(sub_ctx.command.invoke(sub_ctx)) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke airflow-scheduler
15/06/2022, 11:32:18 return ctx.invoke(self.callback, **ctx.params) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke airflow-scheduler
15/06/2022, 11:32:18 return callback(*args, **kwargs) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/cli/params.py", line 23, in decorate airflow-scheduler
15/06/2022, 11:32:18 return func(*args, **kwargs) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/cli/params.py", line 56, in decorate airflow-scheduler
15/06/2022, 11:32:18 func(project, *args, **kwargs) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/cli/invoke.py", line 139, in invoke airflow-scheduler
15/06/2022, 11:32:18 raise invoke_err airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/cli/invoke.py", line 123, in invoke airflow-scheduler
15/06/2022, 11:32:18 exit_code = run_async( airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/core/utils/__init__.py", line 52, in run_async airflow-scheduler
15/06/2022, 11:32:18 loop.run_until_complete(future) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete airflow-scheduler
15/06/2022, 11:32:18 return future.result() airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/cli/invoke.py", line 164, in _invoke airflow-scheduler
15/06/2022, 11:32:18 async with invoker.prepared(session): airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/async_generator/_util.py", line 34, in __aenter__ airflow-scheduler
15/06/2022, 11:32:18 return await self._agen.asend(None) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/core/plugin_invoker.py", line 248, in prepared airflow-scheduler
15/06/2022, 11:32:18 await self.prepare(session) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/core/plugin_invoker.py", line 223, in prepare airflow-scheduler
15/06/2022, 11:32:18 async with self.plugin.trigger_hooks("configure", self, session): airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/async_generator/_util.py", line 34, in __aenter__ airflow-scheduler
15/06/2022, 11:32:18 return await self._agen.asend(None) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/core/behavior/hookable.py", line 87, in trigger_hooks airflow-scheduler
15/06/2022, 11:32:18 await self.__class__.trigger(self, f"before_{hook_name}", *args, **kwargs) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/core/behavior/hookable.py", line 115, in trigger airflow-scheduler
15/06/2022, 11:32:18 raise err airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/core/behavior/hookable.py", line 107, in trigger airflow-scheduler
15/06/2022, 11:32:18 await hook_func(target, *args, **kwargs) airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/core/plugin/airflow.py", line 121, in before_configure airflow-scheduler
15/06/2022, 11:32:18 raise AsyncSubprocessError( airflow-scheduler
15/06/2022, 11:32:18 meltano.core.error.AsyncSubprocessError: Command `airflow --help` failed airflow-scheduler
15/06/2022, 11:32:18 The above exception was the direct cause of the following exception: airflow-scheduler
15/06/2022, 11:32:18 Traceback (most recent call last): airflow-scheduler
15/06/2022, 11:32:18 File "/usr/local/lib/python3.8/site-packages/meltano/cli/__init__.py", line 63, in _run_cli airflow-scheduler
15/06/2022, 11:32:18 raise CliError(str(err)) from err airflow-scheduler
15/06/2022, 11:32:18 meltano.cli.utils.CliError: Command `airflow --help` failed airflow-scheduler
15/06/2022, 11:32:18 Command `airflow --help` failed
Here is a link to the complete slack conversation: https://meltano.slack.com/archives/C01TCRBBJD7/p1655303081378359
Has anyone ever seen this error? My first guess is that it’s linked to not being on airflow 2.X, but would appreciate feedback
Issue Analytics
- State:
- Created a year ago
- Comments:20 (6 by maintainers)
Top Results From Across the Web
Upgrading from 1.10 to 2 - Apache Airflow
No new Airflow 1.x versions will be released. Features in 1.10.15 include: 1. Most breaking DAG and architecture changes of Airflow 2.0 have...
Read more >Add support for Airflow 2.0 (#2558) · Issues - Meltano - GitLab
This is due to Airflow's change of the CLI commands for database initialization. Airflow 1.x uses airflow initdb and 2.x uses the new...
Read more >Move to Airflow 2.0 ADR - Data Ingestion - GitLab
Install Airflow 2.0 to all environments, All CSP's. 2. Airflow 2.0 required DAG code changes. 2.1, Manifest-based ingestion (Python), 5 days ...
Read more >airflow already exist in the db - You.com | The AI Search ...
In mathematics, an idempotent operation is one where f(f(x)) = f(x). ... Update: This bug is fixed in airflow version: 1.10.4. Open side...
Read more >meltano - PyPI
Loaders; DBT Transforms; Meltano Models or .lookml Models; Jupyter Notebook files **; Airflow DAGs as part of an orchestration step **. If you ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@nickhamlin thanks for the write-up! We do call out the new user creation in https://hub.meltano.com/orchestrators/airflow#next-steps and we have a command specified for it in the form of
meltano invoke airflow:create-adminwhich basically does what you say. Agreed though that the migration story isn’t there yet.Thanks everyone! Circling back with a final recap from my end now that the dust has started to settle in hopes of leaving some clues for future searchers who might encounter the same issue.
We’ve got prod up and running again by following a version of @SBurwash 's recommendation and quickly migrating to Airflow 2.3.2. This process requires starting from a blank underlying airflow DB since migrating the old one doesn’t seem to work. There wasn’t really much value in retaining the old airflow data, so starting fresh worked fine (once I figured out the right environment variable names to use described in my comment above).
One other gotcha to be aware of in this process is that Airflow 2 requires a manual CLI command to be run to initialize the first admin user. Airflow 1.X didn’t require this step before allowing access to the GUI, which threw me for a loop. Fortunately, it’s straightforward to fix via a command like
meltano invoke airflow users create --role Admin --username admin --email admin --firstname admin --lastname admin --password admin. Once that runs, this user can then be used to log into the airflow GUI and additional users can be easily managed from there (including removing this temp admin user for clear security reasons).That’s all the loose ends I can think of, but I’ll come back and edit this comment if I come across others.