Duplicated (non-default) list entries when using environments and merge enabled

First and foremost: Thank you for this project

I stumbled over an issue with environments and the merge of lists, namely non-default list items are duplicated in final (environment-specific) list.

Steps to reproduce

python==3.9.6 dynaconf==3.1.5

config file: /some_folder/test_settings.yaml

default:
  SOME_KEY: "value"
  SOME_LIST:
    - "item_1"
    - "item_2"
    - "item_3"
other:
  SOME_KEY: "new_value"
  SOME_LIST:
    - "item_4"
    - "item_5"

test file: /some_folder/test.py

from dynaconf import Dynaconf

settings = Dynaconf(settings_files=["test_settings.yaml"], environments=True, merge_enabled=True)

print(f'default env: {settings.from_env("default").SOME_LIST}')
print(f'other env: {settings.from_env("other").SOME_LIST}')

results in:

default env: ['item_1', 'item_2', 'item_3']
other env: ['item_1', 'item_2', 'item_3', 'item_4', 'item_5', 'item_4', 'item_5']

expected:

default env: ['item_1', 'item_2', 'item_3']
other env: ['item_1', 'item_2', 'item_3', 'item_4', 'item_5']

[rochacbruno]

@andressadotpy yes, a simpler example is also listed on https://www.dynaconf.com/merging/#avoiding-duplications-on-lists

given existing setting is

In [1]: from dynaconf import Dynaconf

In [2]: settings = Dynaconf(
   ...:   SCRIPTS=["1.sh", "2.sh", "3.sh"]
   ...: )

In [3]: settings.scripts
Out[3]: <BoxList: ['1.sh', '2.sh', '3.sh']>

Now assume the user wants to add ['3.sh', '4.sh'] to the list and get no duplications

In [4]: settings.set("SCRIPTS", ["3.sh", "4.sh", "dynaconf_merge_unique"])

In [5]: settings.scripts
Out[5]: <BoxList: ['1.sh', '2.sh', '3.sh', '4.sh']>

But, there is another case when the user really wants to duplicate, for example, add "1.sh" again at the end.

In [6]: settings.set("SCRIPTS", "@merge 1.sh")

In [7]: settings.scripts
Out[7]: <BoxList: ['1.sh', '2.sh', '3.sh', '4.sh', '1.sh']>

The Problem is that, wherever the data comes from:

• environment variables
• settings files in multiple envs default, other, etc
• settings.set calls

The same object_merge function is called, and looks like we need a different behavior for this issue specific case.

The problem described here can be solved if the settings file is

default:
  SOME_KEY: "value"
  SOME_LIST:
    - "item_1"
    - "item_2"
    - "item_3"
other:
  SOME_KEY: "new_value"
  SOME_LIST:
    - "dynaconf_merge_unique"
    - "item_4"
    - "item_5"

Notice: “dynaconf_merge_unique” as part of the list.

We maybe need to implicit add that mark on a list, if the key exist in an environment called default

[andressadotpy]

To solve this issue based in this discussion I thought about a small change inside dynaconf/base.py _merge_before_set() method.

        if isinstance(value, (list, tuple)):
            local_merge = (
                "dynaconf_merge" in value or "dynaconf_merge_unique" in value
            )
            default_env = self.DEFAULT_ENV_FOR_DYNACONF == 'DEFAULT'  # This is different
            if global_merge or local_merge:
                value = list(value)
                unique = True if default_env else False  # This is different
                if local_merge:
                    try:
                        value.remove("dynaconf_merge")
                    except ValueError:  # EAFP
                        value.remove("dynaconf_merge_unique")
                        unique = True
                value = object_merge(existing, value, unique=unique)
        return value

All tests passed with those changes, including the test I added with this problem. What do you think about those changes?