Add from Hub cant find the proper file bundle

Using the main branch locally I’m getting this error message when trying to add Airflow, it looks like it cant find the associated file bundle.

cc @edgarrmondragon @tayloramurphy

Patricks-MacBook-Pro:meltano pnadolny$ poetry run meltano --log-level=debug add orchestrator airflow
2022-06-02T14:33:33.341757Z [info     ] Environment 'dev' is active
2022-06-02T14:33:33.378071Z [debug    ] Creating engine <meltano.core.project.Project object at 0x108cdce50>@sqlite:////Users/pnadolny/Documents/Git/GitHub/pnadolny/meltano_example_implementations/meltano_projects/end_to_end_demo/.meltano/meltano.db
2022-06-02T14:33:33.525444Z [debug    ] Lockfile is feature-flagged    status=False
2022-06-02T14:33:33.533284Z [debug    ] Starting new HTTPS connection (1): hub.meltano.com:443
2022-06-02T14:33:33.761060Z [debug    ] https://hub.meltano.com:443 "GET /meltano/api/v1/plugins/orchestrators/index HTTP/1.1" 200 225
2022-06-02T14:33:33.890847Z [debug    ] https://hub.meltano.com:443 "GET /meltano/api/v1/plugins/orchestrators/airflow--apache HTTP/1.1" 200 982
2022-06-02T14:33:33.948498Z [debug    ] Found plugin                   plugin=airflow source=hub
Added orchestrator 'airflow' to your Meltano project
Variant:        apache (default)
Repository:     https://github.com/apache/airflow
Documentation:  https://docs.meltano.com/guide/orchestration

2022-06-02T14:33:34.090314Z [debug    ] Starting new HTTPS connection (1): discovery.meltano.com:443
2022-06-02T14:33:34.176982Z [debug    ] https://discovery.meltano.com:443 "GET /discovery.yml HTTP/1.1" 200 23849
2022-06-02T14:33:34.729529Z [error    ] The `discovery.yml` manifest received from https://discovery.meltano.com/discovery.yml could not be parsed.
2022-06-02T14:33:34.729662Z [debug    ] 'models' is not a valid PluginType
2022-06-02T14:33:34.730883Z [warning  ] Falling back on the cached `discovery.yml` manifest...
2022-06-02T14:33:34.731484Z [warning  ] Falling back on the bundled `discovery.yml` manifest...
2022-06-02T14:33:35.521898Z [debug    ] Lockfile is feature-flagged    status=False
2022-06-02T14:33:35.641751Z [debug    ] https://hub.meltano.com:443 "GET /meltano/api/v1/plugins/files/index HTTP/1.1" 200 1541
2022-06-02T14:33:35.759144Z [debug    ] https://hub.meltano.com:443 "GET /meltano/api/v1/plugins/files/airflow--meltano HTTP/1.1" 200 285
2022-06-02T14:33:35.760108Z [debug    ] Found plugin                   plugin=airflow source=hub
2022-06-02T14:33:35.775057Z [debug    ] Lockfile is feature-flagged    status=False
2022-06-02T14:33:35.864715Z [debug    ] https://hub.meltano.com:443 "GET /meltano/api/v1/plugins/files/index HTTP/1.1" 200 1541
2022-06-02T14:33:35.865428Z [debug    ] File bundle 'airflow' variant 'original' is not known to Meltano. Variants: ['meltano (default)'] 
Traceback (most recent call last):
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/hub/client.py", line 171, in find_definition
    url = plugin.variants[variant_name].ref
KeyError: 'original'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/cli/__init__.py", line 51, in main
    cli(obj={"project": None})
  File "/Users/pnadolny/Library/Caches/pypoetry/virtualenvs/meltano-DNTywUpc-py3.9/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/pnadolny/Library/Caches/pypoetry/virtualenvs/meltano-DNTywUpc-py3.9/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/pnadolny/Library/Caches/pypoetry/virtualenvs/meltano-DNTywUpc-py3.9/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/pnadolny/Library/Caches/pypoetry/virtualenvs/meltano-DNTywUpc-py3.9/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/pnadolny/Library/Caches/pypoetry/virtualenvs/meltano-DNTywUpc-py3.9/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/cli/params.py", line 23, in decorate
    return func(*args, **kwargs)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/cli/params.py", line 56, in decorate
    func(project, *args, **kwargs)
  File "/Users/pnadolny/Library/Caches/pypoetry/virtualenvs/meltano-DNTywUpc-py3.9/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/cli/add.py", line 115, in add
    related_plugins = add_related_plugins(
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/cli/utils.py", line 383, in add_related_plugins
    related_plugins = add_service.add_related(
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/project_add_service.py", line 103, in add_related
    plugin = self.add(plugin_ref.type, plugin_ref.name)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/project_add_service.py", line 63, in add
    added = self.add_plugin(plugin)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/project_add_service.py", line 82, in add_plugin
    return self.plugins_service.add_to_file(plugin)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/project_plugins_service.py", line 153, in add_to_file
    existing_plugin = self.get_plugin(plugin)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/project_plugins_service.py", line 317, in get_plugin
    return self.ensure_parent(plugin)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/project_plugins_service.py", line 559, in ensure_parent
    plugin.parent = self.get_parent(plugin)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/project_plugins_service.py", line 544, in get_parent
    parent, source = self.find_parent(plugin)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/project_plugins_service.py", line 533, in find_parent
    raise error
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/project_plugins_service.py", line 528, in find_parent
    return (self._get_parent_from_hub(plugin), DefinitionSource.HUB)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/project_plugins_service.py", line 465, in _get_parent_from_hub
    return self.hub_service.get_base_plugin(plugin, variant_name=plugin.variant)
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/plugin_discovery_service.py", line 136, in get_base_plugin
    plugin = project_plugin.custom_definition or self.find_definition(
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/core/hub/client.py", line 173, in find_definition
    raise HubPluginVariantNotFound(
meltano.core.hub.client.HubPluginVariantNotFound: File bundle 'airflow' variant 'original' is not known to Meltano. Variants: ['meltano (default)']

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/pnadolny/Documents/Git/GitHub/meltano/meltano/src/meltano/cli/__init__.py", line 59, in main
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: File bundle 'airflow' variant 'original' is not known to Meltano. Variants: ['meltano (default)']
File bundle 'airflow' variant 'original' is not known to Meltano. Variants: ['meltano (default)']

[edgarrmondragon]

@tayloramurphy #5963 should close it too, but we may be blocked by https://github.com/meltano/meltano/pull/5957/

[tayloramurphy]

@edgarrmondragon ok, so I removed the plugin (see #5979 for more details) and then re-added with the debug flag and I do see it added from the Hub.

Maybe I got my install into a weird state b/c now I’m getting KeyError: 'venv_name'

Full log lines

❯ meltano --log-level=debug add orchestrator airflow 2022-06-02T15:22:40.757301Z [info ] Environment 'dev' is active 2022-06-02T15:22:40.796019Z [debug ] Creating engine <meltano.core.project.Project object at 0x110a87be0>@sqlite:////Users/taylormurphy/Documents/Projects/dev/meltano/addfromhub/.meltano/meltano.db 2022-06-02T15:22:41.171230Z [debug ] Lockfile is feature-flagged status=True 2022-06-02T15:22:41.244429Z [debug ] Starting new HTTPS connection (1): hub.meltano.com:443 2022-06-02T15:22:41.589888Z [debug ] https://hub.meltano.com:443 "GET /meltano/api/v1/plugins/orchestrators/index HTTP/1.1" 200 225 2022-06-02T15:22:41.774020Z [debug ] https://hub.meltano.com:443 "GET /meltano/api/v1/plugins/orchestrators/airflow--apache HTTP/1.1" 200 982 2022-06-02T15:22:41.776976Z [debug ] Found plugin plugin=airflow source=hub 2022-06-02T15:22:41.845958Z [info ] Locking a <class 'meltano.core.plugin.airflow.Airflow'> Plugin definition is already locked at /Users/taylormurphy/Documents/Projects/dev/meltano/addfromhub/plugins/orchestrators/airflow--apache.lock. You can remove the file manually to avoid using a stale definition.

2022-06-02T15:22:41.864439Z [debug ] Starting new HTTPS connection (1): www.google-analytics.com:443 2022-06-02T15:22:41.960329Z [debug ] https://www.google-analytics.com:443 “POST /collect HTTP/1.1” 200 35 2022-06-02T15:22:41.994755Z [debug ] Starting new HTTPS connection (1): sp.meltano.com:443 2022-06-02T15:22:42.791291Z [debug ] https://sp.meltano.com:443 “GET /i?e=se&se_ca=meltano+add+orchestrators&se_ac=meltano+add+orchestrators+airflow±-environment%3Def260e9aa3c673af240d17a2660480361a8e081d1ffeca2a5ed0e3219fc18567&se_la=7148ed44-bfdf-4a56-baa8-41e9aef53d43&eid=7b5ef1a7-4976-4cae-a1b7-3ce606db6f1a&dtm=1654183361961&tv=py-0.10.0&p=pc&stm=1654183361000 HTTP/1.1” 302 0 2022-06-02T15:22:42.906524Z [debug ] https://sp.meltano.com:443 “GET /i?e=se&eid=7b5ef1a7-4976-4cae-a1b7-3ce606db6f1a&stm=1654183361000&tv=py-0.10.0&se_ac=meltano+add+orchestrators+airflow±-environment%3Def260e9aa3c673af240d17a2660480361a8e081d1ffeca2a5ed0e3219fc18567&se_la=7148ed44-bfdf-4a56-baa8-41e9aef53d43&se_ca=meltano+add+orchestrators&n3pc=true&p=pc&dtm=1654183361961 HTTP/1.1” 200 43 2022-06-02T15:22:43.137601Z [debug ] Starting new HTTPS connection (1): discovery.meltano.com:443 2022-06-02T15:22:43.298983Z [debug ] https://discovery.meltano.com:443 “GET /discovery.yml?project_id=7148ed44-bfdf-4a56-baa8-41e9aef53d43 HTTP/1.1” 200 23849 2022-06-02T15:22:44.106455Z [error ] The discovery.yml manifest received from https://discovery.meltano.com/discovery.yml could not be parsed. 2022-06-02T15:22:44.106568Z [debug ] ‘models’ is not a valid PluginType 2022-06-02T15:22:44.107514Z [warning ] Falling back on the cached discovery.yml manifest… 2022-06-02T15:22:44.761035Z [debug ] Lockfile is feature-flagged status=True 2022-06-02T15:22:44.898841Z [debug ] https://hub.meltano.com:443 “GET /meltano/api/v1/plugins/files/index HTTP/1.1” 200 1541 2022-06-02T15:22:45.022899Z [debug ] https://hub.meltano.com:443 “GET /meltano/api/v1/plugins/files/airflow–meltano HTTP/1.1” 200 285 2022-06-02T15:22:45.023478Z [debug ] Found plugin plugin=airflow source=hub 2022-06-02T15:22:45.028634Z [debug ] Found plugin plugin=airflow source=lockfile 2022-06-02T15:22:45.029154Z [debug ] Using selector: KqueueSelector 2022-06-02T15:22:45.029404Z [debug ] venv_name
Traceback (most recent call last): File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/core/behavior/canonical.py”, line 138, in getattr value = self._dict[attr] KeyError: ‘venv_name’

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/core/plugin/base.py”, line 496, in getattr return getattr(self._plugin_def, attr) File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/core/behavior/canonical.py”, line 143, in getattr raise AttributeError(attr) from err AttributeError: venv_name

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/core/behavior/canonical.py”, line 138, in getattr value = self._dict[attr] KeyError: ‘venv_name’

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/cli/init.py”, line 51, in main cli(obj={“project”: None}) File “/Users/taylormurphy/Library/Caches/pypoetry/virtualenvs/meltano-AqYjSFXr-py3.8/lib/python3.8/site-packages/click/core.py”, line 829, in call return self.main(*args, **kwargs) File “/Users/taylormurphy/Library/Caches/pypoetry/virtualenvs/meltano-AqYjSFXr-py3.8/lib/python3.8/site-packages/click/core.py”, line 782, in main rv = self.invoke(ctx) File “/Users/taylormurphy/Library/Caches/pypoetry/virtualenvs/meltano-AqYjSFXr-py3.8/lib/python3.8/site-packages/click/core.py”, line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File “/Users/taylormurphy/Library/Caches/pypoetry/virtualenvs/meltano-AqYjSFXr-py3.8/lib/python3.8/site-packages/click/core.py”, line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File “/Users/taylormurphy/Library/Caches/pypoetry/virtualenvs/meltano-AqYjSFXr-py3.8/lib/python3.8/site-packages/click/core.py”, line 610, in invoke return callback(*args, **kwargs) File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/cli/params.py”, line 23, in decorate return func(*args, **kwargs) File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/cli/params.py”, line 56, in decorate func(project, *args, **kwargs) File “/Users/taylormurphy/Library/Caches/pypoetry/virtualenvs/meltano-AqYjSFXr-py3.8/lib/python3.8/site-packages/click/decorators.py”, line 21, in new_func return f(get_current_context(), *args, **kwargs) File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/cli/add.py”, line 128, in add success = install_plugins( File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/cli/utils.py”, line 426, in install_plugins install_results = install_service.install_plugins(plugins, reason=reason) File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/core/plugin_install_service.py”, line 251, in install_plugins states, new_plugins = self.remove_duplicates(plugins=plugins, reason=reason) File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/core/plugin_install_service.py”, line 202, in remove_duplicates if (plugin.type, plugin.venv_name) not in seen_venvs: File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/core/plugin/base.py”, line 498, in getattr return getattr(self._variant, attr) File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/core/behavior/canonical.py”, line 143, in getattr raise AttributeError(attr) from err AttributeError: venv_name

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “/Users/taylormurphy/Documents/Projects/dev/meltano/src/meltano/cli/init.py”, line 59, in main raise CliError(str(err)) from err meltano.cli.utils.CliError: venv_name venv_name