Custom modules synced after pillar render

Description of Issue

Custom modules, are synced either via manual saltutil.sync_all or state.highstate run.
During highstate run they are synced after the pillar renders. However the highstates pillar may already be using them and thus will fail.

1.Is it something that may be relaxed to allow usage of custom modules during the very first highstate run?

2.Why does the pillar render before the sync_all? To me It seems that first syncing the modules then rendering is more natural way.

This is a bit unwieldy, especially when using salt provisioner for Vagrant

Please see the below setup instruction.

Setup

Any setup with custom module: e.g: _sdb/kdbx.py

Steps to Reproduce Issue

1. Clean Salt install (masterless will do)
2. Custom module in _sdb (in State Tree Root)
3. Pillar data that already uses this module: somepillar.sls

key: {{ salt['sdb.get']("sdb://kdbx/secret/path") }}

1. Run state.highstate

The 4. will fail with:

[CRITICAL] Rendering SLS 'REDACTED' failed, render error:                                                                                                                                                                                                                                                                    
Jinja error: 'kdbx.get'                                                                                                                                                                                                                                                                                                      
Traceback (most recent call last):                                                                                                                                                                                                                                                                                           
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl                                                                                                                                                                                                                              
    output = template.render(**decoded_context)                                                                                                                                                                                                                                                                              
  File "/usr/local/lib/python3.5/dist-packages/jinja2/environment.py", line 1008, in render                                                                                                              
    return self.environment.handle_exception(exc_info, True)                                                                                
  File "/usr/local/lib/python3.5/dist-packages/jinja2/environment.py", line 780, in handle_exception                                                    
    reraise(exc_type, exc_value, tb)                                                                                                                                                        
  File "/usr/local/lib/python3.5/dist-packages/jinja2/_compat.py", line 37, in reraise                                                                                                                                  
    raise value.with_traceback(tb)                                                                                                        
  File "<template>", line 122, in top-level template code                                                                                                                                                                                                                                                                    
  File "/usr/lib/python3/dist-packages/salt/modules/sdb.py", line 28, in get                                                                                                                                                                                                                                                 
    return salt.utils.sdb.sdb_get(uri, __opts__, __utils__)                                                                                                                                                                                                                                                                  
  File "/usr/lib/python3/dist-packages/salt/utils/sdb.py", line 46, in sdb_get                                                                                                                                                                                                                                               
    return loaded_db[fun](query, profile=profile)                                                                                                                            
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1208, in __getitem__                                                                   
    func = super(LazyLoader, self).__getitem__(item)                                                                                                                                                                                                                            
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__                                                                  
    raise KeyError(key)                                                                                                                                                  
KeyError: 'kdbx.get'                   

Versions Report

Salt Version:
           Salt: 2019.2.2
 
Dependency Versions:
           cffi: 1.13.2
       cherrypy: 18.4.0
       dateutil: 2.8.1
      docker-py: 1.10.6
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.3
        libgit2: 0.28.2
        libnacl: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 3.9.4
   pycryptodome: Not Installed
         pygit2: 0.28.2
         Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
   python-gnupg: Not Installed
         PyYAML: 5.1.2
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.1 
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-8-amd64
         system: Linux
        version: debian 9.1 

[kiemlicz]

Originally I wanted to use this in kitchen-salt, because the: https://github.com/saltstack/kitchen-salt/issues/295 was open back then
However the kitchen-salt introduced the pre_salt_command which is a workaround for me

I’m not aware of any workaround for this issue here, other than manually running sync_all prior to highstate - absolutely terrible…

[jkenn99]

@kiemlicz have you come up with any workarounds for this?