settings.set() does not update dynamic config values

Hi all,

In brief:

I have a script that monitors Redis sentinels and periodically updates the available master and slaves through dynaconf. These values are accessed by other python scripts to optimise specific read/write cases.

The issue:

calling settings.set('keyname', python_dict_obj) or settings.set('keyname', json.dumps(python_dict_obj)) does not update any values in config.toml nor can they be accessed from other python scripts.

Reproducing the problem

Environment variables that I set in all open terminals: export ENV_FOR_DYNACONF=development export LANG=en_US.UTF-8 export LC_ALL=en_US.UTF-8

Initialization of settings dynaconf init -e development - this generates settings.toml with dummy values in five sections [default], [development], [staging], [production], [global]

Main python script (irrelevant parts omitted for clarity)

from redis.sentinel import Sentinel
import redis
import json
import time
import sys
from dynaconf import settings


REDIS_SENTINELS = [("127.0.0.1", 16380), ("127.0.0.1", 16381), ("127.0.0.1", 16382)]
REDIS_CLUSTER_NAME = 'redis-cluster'

_sentinel = Sentinel(sentinels=REDIS_SENTINELS, min_other_sentinels=1, socket_timeout=0.1)


def _set_master():
    _master = _sentinel.discover_master(service_name=REDIS_CLUSTER_NAME)
    # settings.set('master', json.dumps(list(_master)))
    print("Got master ", _master)
    settings.set('master', _master)
    print(settings.exists('MASTER'))
    print(settings.MASTER)


def _set_slaves():
    _slaves = _sentinel.discover_slaves(service_name=REDIS_CLUSTER_NAME)
    # settings.set('slaves', json.dumps(_slaves))
    print("got slaves ", _slaves)
    settings.set('slaves', _slaves)
    print(settings.exists('SLAVES'))
    print(settings.SLAVES)


def main():
    while True:
        _set_master()
        _set_slaves()
        time.sleep(10)


if __name__ == '__main__':
    main()

The output

As you notice, the settings.exists() shows the values being set as True and also I can access the values

Got master  ('127.0.0.1', 6380)
True
['127.0.0.1', 6380]
got slaves  [('127.0.0.1', 6382), ('127.0.0.1', 6381)]
True
[['127.0.0.1', 6382], ['127.0.0.1', 6381]]

When I open another terminal and run a different script or the interactive interpreter export ENV_FOR_DYNACONF=development export LANG=en_US.UTF-8 export LC_ALL=en_US.UTF-8

$python
>>> from dynaconf import settings
>>> settings.MASTER
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/anomit/.venv-py3/lib/python3.6/site-packages/dynaconf/base.py", line 100, in __getattr__
    return getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'MASTER'

Where am I going wrong?

[rochacbruno]

Hi @anomit

The settings.set method is meant to set a value on the in-memory settings object, this method exists to read the values from sources like a toml file or a redis server and then set it to the efemeral settings object.

This method IS NOT to change the settings file or its sources.

For your use case I recommend 2 options:

1. Option 1Use the write method of specific loader.

from dynaconf.loaders.toml_loader import write
write('path/to/file.toml', {"default": {"foo": 123}}, merge=True)

With that call you will have foo = 123 added to the [default] section of the settings file. https://github.com/rochacbruno/dynaconf/blob/master/dynaconf/loaders/toml_loader.py#L39

WARNING: Writing to settings file is not thread safe you may have data races if running multiple processes writing to the same file.

1. Option 2 is using the REDIS loader (recommended)

If you want to have your settings dynamic and mutable you may use an external server like REDIS take a look at https://dynaconf.readthedocs.io/en/latest/guides/external_storages.html#using-redis that way you do not have to read the settings.toml file and all your settings can live in a redis server, easier to read and write, you can also use the dynaconf.loaders.redis_loader.write method to write to it.

Other options includes implementing your custom loader or using other external service like vaultProject.

[anomit]

That’s alright. I changed to a json settings file.

The interesting behavior was that on setting merge=True , if the value corresponding to the key is a list or nested lists, dynaconf appends the elements.

For example, from the code aboce: 1st run of the loop: {"development": "master": ["127.0.0.1", 6380]} 2nd run: {"development": "master": ["127.0.0.1", 6380, "127.0.0.1", 6380]} …and so on

Is this behavior by design? Or is there a scope for introducing a flag to modify this behavior?