Performance concerns

Hi there,

I have a project that uses dynaconf v2.2.3 (thanks for making dynaconf, btw, nice project !) and I would like to discuss two distincts issues:

A. Environment switching seems very costly

I use dynaconf for a data engineering project, where I use it to configure several data connectors. I have a routine that validates my connector’s configuration globally on all environments. The simplest way to implement it for me is to iterate through each connector, and for each of them, validate the configuration of that connector against each environment (which I do by using settings.setenv() to switch environment temporarily)

I noticed that this was agonizingly slow. It looks like each time I call settings.setenv(), it takes 0.2 seconds, which quickly adds up. For instance, my validation routine which took 3.5 seconds in total and takes several minutes if I use settings.setenv() before every check.

Would it be possible to add to dynaconf some mechanism that avoids reloading everything when switching to an env that was already loaded ?

B. dynaconf 3 seems seems much slower than dynaconf 2

I tried switching to dynaconf v3.1.5 to see if perhaps the latest version already had improvements for my issue above. When I ran my routine (without the settings.setenv() trick) it took 3.5 sec in v2.2.3 and it took 8.7 seconds in v3.1.5, which is nearly 3 times slower.

I did not change a single line of code besides the dynaconf version change. Perhaps did I miss something that I was supposed to change in the way I call dynaconf ? I don’t do anything particularly fancy with dynaconf, mostly import settings and settings.get(). The only relevant detail is that I have a huge settings.toml file (~2500 lines for 5 environments).

If it might help, below is the summary I got with cProfile:

With version 2.2.3

    8 559 695 function calls (8490226 primitive calls) in 3.200 seconds
   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   2063/1    0.008    0.000    3.202    3.202 {built-in method builtins.exec}
        1    0.000    0.000    3.201    3.201 <string>:1(<module>)
        1    0.000    0.000    3.201    3.201 launcher.py:25(main)
        1    0.000    0.000    2.826    2.826 validate.py:142(do_command)
        1    0.009    0.009    2.361    2.361 validate.py:79(_validate_populates)
     1346    0.006    0.000    1.264    0.001 load.py:169(__load_file)
     1320    0.002    0.000    1.261    0.001 exec.py:18(load_populate)
     1320    0.002    0.000    1.259    0.001 exec.py:12(__load_populate)
     1346    0.004    0.000    1.258    0.001 load.py:162(__load_job_module)
23723/23563    0.018    0.000    0.982    0.000 parse_conf.py:79(evaluate)
     1346    0.002    0.000    0.813    0.001 connection_validations.py:106(validate_connections)
     1346    0.002    0.000    0.812    0.001 connection_validations.py:82(__validate_connections_on_env)
     1346    0.004    0.000    0.798    0.001 connection_validations.py:58(__validate_connections)
      569    0.002    0.000    0.793    0.001 connection_validations.py:41(load_and_validate_external_io)
      569    0.002    0.000    0.787    0.001 connector.py:80(load_connector)
     1346    0.004    0.000    0.701    0.001 load.py:65(__delete_template_modules)
     1346    0.361    0.000    0.681    0.001 load.py:74(<listcomp>)
     5167    0.004    0.000    0.665    0.000 file_index.py:34(__index_files_with_name)
        3    0.004    0.001    0.605    0.202 file_index.py:18(__index_files_with_name_cached)
2421/1376    0.008    0.000    0.600    0.000 <frozen importlib._bootstrap>:651(_load_unlocked)
     4077    0.019    0.000    0.569    0.000 file_index.py:46(list_schema_table_folders)
2046/1376    0.004    0.000    0.560    0.000 <frozen importlib._bootstrap_external>:672(exec_module)
   108157    0.030    0.000    0.540    0.000 {built-in method builtins.getattr}
     1346    0.012    0.000    0.530    0.000 load.py:79(__load_module_file)
      865    0.003    0.000    0.503    0.001 boxing.py:10(__getattr__)
     1730    0.008    0.000    0.500    0.000 box.py:503(__getattr__)
      569    0.002    0.000    0.499    0.001 __init__.py:77(get_env_conf)
2564/1377    0.001    0.000    0.485    0.000 <frozen importlib._bootstrap>:211(_call_with_frames_removed)

With version 3.1.5

    31 090 566 function calls (29705073 primitive calls) in 12.231 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   2063/1    0.007    0.000   12.233   12.233 {built-in method builtins.exec}
        1    0.000    0.000   12.233   12.233 <string>:1(<module>)
        1    0.000    0.000   12.233   12.233 launcher.py:25(main)
        1    0.000    0.000   11.796   11.796 validate.py:142(do_command)
        1    0.010    0.010   11.288   11.288 validate.py:79(_validate_populates)
1068565/26898    0.776    0.000    9.448    0.000 boxing.py:14(evaluate)
    11282    0.045    0.000    9.309    0.001 base.py:410(get)
    25443    0.061    0.000    9.227    0.000 boxing.py:68(get)
    11199    0.017    0.000    8.955    0.001 boxing.py:61(_case_insensitive_get)
    11201    0.011    0.000    8.770    0.001 box.py:119(items)
    11201    0.217    0.000    8.752    0.001 box.py:119(<listcomp>)
     1320    0.002    0.000    7.051    0.005 exec.py:18(load_populate)
     1320    0.002    0.000    7.049    0.005 exec.py:12(__load_populate)
1672333/1672321    0.689    0.000    6.834    0.000 {built-in method builtins.getattr}
   133928    0.264    0.000    6.000    0.000 boxing.py:33(__getattr__)
     1346    0.007    0.000    5.957    0.004 load.py:169(__load_file)
     1346    0.004    0.000    5.950    0.004 load.py:162(__load_job_module)
   267856    0.667    0.000    5.643    0.000 box.py:165(__getattr__)
     5167    0.005    0.000    5.091    0.001 file_index.py:34(__index_files_with_name)
     5139    0.002    0.000    4.678    0.001 file_index.py:61(__get_indexed_populates)
     5167    0.009    0.000    4.496    0.001 __init__.py:49(enable_file_index_cache)
     5138    0.005    0.000    4.485    0.001 file_index.py:128(get_populate)
921207/666486    0.416    0.000    4.302    0.000 __init__.py:349(recursively_evaluate_lazy_format)
2421/1376    0.008    0.000    4.019    0.003 <frozen importlib._bootstrap>:651(_load_unlocked)
     1346    0.013    0.000    3.994    0.003 load.py:79(__load_module_file)
2046/1376    0.004    0.000    3.979    0.003 <frozen importlib._bootstrap_external>:672(exec_module)
6141/1711    0.005    0.000    3.914    0.002 <frozen importlib._bootstrap_external>:393(_check_name_wrapper)
     1365    0.002    0.000    3.912    0.003 <frozen importlib._bootstrap_external>:813(load_module)
     1365    0.001    0.000    3.911    0.003 <frozen importlib._bootstrap_external>:680(load_module)
     1365    0.002    0.000    3.910    0.003 <frozen importlib._bootstrap>:253(_load_module_shim)
2564/1377    0.001    0.000    3.902    0.003 <frozen importlib._bootstrap>:211(_call_with_frames_removed)

I believe that the files launcher.py, validate.py, exec.py, load.py and file_index.py are mine, but as you can see what really changes between version 2 and 3 is this line

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
- 23723/23563    0.018    0.000    0.982    0.000 parse_conf.py:79(evaluate)
+ 1068565/26898    0.776    0.000    9.448    0.000 boxing.py:14(evaluate)

The number of calls made to the evaluate() function jumped from 23,563 to 1,068,565, mostly recursive calls (the number of root calls only being 26,898). Surely there is something sub-optimal happening somewhere.

Please tell me if you need me to provide more information or run more tests.

[FurcyPin]

I don’t know, from_env uses a cache called _env_cache, so reloading an environment that has already been loaded costs nothing. On the other hand, setenv doesn’t seem to use that cache at all, and seems to reload everything each time.

I didn’t try to set clean=False while calling setenv. I could try that too. But I’m not sure about what that clean option is meant to do exactly. It’s doc sounds like it is meant to remove variables from the first env when they are not overridden by the new env. If so, this is not what I need.

I’m fine with using from_env anyway. The functional approach is often cleaner than the imperative one.

P.S. the name from_env sounds more suitable to me than .clone or .copy, given what it does. Maybe get_settings_for_env ?

[FurcyPin]

I confirm, using from_env instead of setenv solved my first issue like a charm.