bug: Regression in `meltano remove`

Meltano Version

2.3.0

Python Version

NA

Bug scope

CLI (options, error messages, logging, etc.)

Operating System

NA

Description

Meltano 2.3.0 introduced a regression on meltano remove where it complains that the plugin is missing the settings_with_extras attribute.

The regression was most likely introduced in https://github.com/meltano/meltano/pull/6267 somewhere in the use of a feature flag:

https://github.com/meltano/meltano/blob/23a9099cefe148cb560d25e7ae335447858948df/src/meltano/core/plugin/settings_service.py#L63-L65

I still haven’t figured out the root cause but might be related with trying to check for feature flags in the PluginSettingsService.

Other commands may be similarly affected, but at least invoke and run seem to be working fine.

Code

Traceback

Traceback (most recent call last):
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/behavior/canonical.py", line 138, in __getattr__
    value = self._dict[attr]
KeyError: 'settings_with_extras'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/__init__.py", line 66, in _run_cli
    cli(obj={"project": None})
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/cli.py", line 35, in main
    return super().main(*args, windows_expand_args=False, **kwargs)
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/utils.py", line 540, in invoke
    super().invoke(ctx)  # noqa: WPS608
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/params.py", line 18, in decorate
    return func(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/params.py", line 53, in decorate
    func(project, *args, **kwargs)
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/meltano-SY7IjDqw-py3.9/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/remove.py", line 32, in remove
    remove_plugins(project, plugins)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/remove.py", line 39, in remove_plugins
    num_removed, total = remove_service.remove_plugins(
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin_remove_service.py", line 60, in remove_plugins
    removal_managers = self.remove_plugin(plugin)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin_remove_service.py", line 87, in remove_plugin
    DbRemoveManager(plugin, self.project),
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin_location_remove.py", line 86, in __init__
    self.plugins_settings_service = PluginSettingsService(project, plugin)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin/settings_service.py", line 63, in __init__
    with self.feature_flag(
  File "/Users/edgarramirez/.pyenv/versions/3.9.13/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 657, in feature_flag
    allowed = self.get(f"{FEATURE_FLAG_PREFIX}.{feature}") or False
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 444, in get
    value, _ = self.get_with_source(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 431, in get_with_source
    value, metadata = self.get_with_metadata(*args, **kwargs)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 333, in get_with_metadata
    setting_def = setting_def or self.find_setting(name)
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 598, in find_setting
    for setting in self.definitions()
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/settings_service.py", line 568, in definitions
    for setting in self.setting_definitions
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/plugin/settings_service.py", line 136, in setting_definitions
    settings = self.plugin.settings_with_extras
  File "/Users/edgarramirez/meltano/meltano/src/meltano/core/behavior/canonical.py", line 143, in __getattr__
    raise AttributeError(attr) from err
AttributeError: settings_with_extras

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/edgarramirez/meltano/meltano/src/meltano/cli/__init__.py", line 74, in _run_cli
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: settings_with_extras
settings_with_extras

[edgarrmondragon]

@kgpayne a good way to prevent future regressions would be to add a meltano remove at the end of this integration test:

https://github.com/meltano/meltano/blob/40a8066c3d3b5ad403bc08a4e51d195ffdd1e02a/docs/example-library/meltano-objects/index.md?plain=1#L1

[edgarrmondragon]

I’m realizing that it’s not very intuitive to have access to the feature_flag method in the SettingsService base class. I’m not sure what all changes it would entail, but it might be worth moving that method into ProjectSettingsService only, if possible to avoid this in the future.

@cjohnhanson Thanks for jumping in! Yeah, it makes sense to implement feature flags exclusively at the project settings level. I’m almost sure it wouldn’t cause any issues.