Nested includes fail when addressing the state with '/' instead of '.'

Description of Issue/Question

I have a state directory containing various tools for our multi-tier application. Each tier has its own sls file that includes a handful of states that are specific to that tier. All tier-SLS-files are included by init.sls and chosen by matching grains that identify the tiers.

If I apply the state directory (and thus the init.sls), everything works fine. If I apply one of the tier-SLS alone, includes fail if I address the tier-SLS with ‘/’ in the path. If I use ‘.’ instead, it works.

Difficult to describe, but testcase shown below should make it clear

Testcase-Files below…

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

All in directory ‘testcase’ below the salt-root:

---------- testcase/init.sls ----------

include:
    - .substate

---------- testcase/substate.sls ----------

include:
    - .ze-file

---------- testcase/ze-file.sls ----------

/tmp/test.dat:
    file.managed:
        - source: salt://{{slspath}}/test.dat

---------- testcase/test.dat ----------

Hello, world

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

–> Debug log down below the version report…

salt $HOSTNAME state.apply testcase <-- works salt $HOSTNAME state.apply testcase/ze-file <-- works salt $HOSTNAME state.apply testcase/substate <-- fails salt $HOSTNAME state.apply testcase.substate <-- works

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Master and minion are the same host for this specific test, so no differences in version.

salt:/srv/salt/testcase# salt --versions-report
Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 4.20.0-1.el7.elrepo.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

Debug log for failing salt $HOSTNAME state.apply testcase/substate

[INFO    ] User sudo_michael Executing command state.apply with jid 20190118134335325608
[DEBUG   ] Command details {u'tgt_type': u'glob', u'jid': u'20190118134335325608', u'tgt': u'salt', u'ret': u'', u'user': u'sudo_michael', u'arg': [u'testcase/substate'], u'fun': u'state.apply'}
[INFO    ] Starting a new job with PID 3017
[DEBUG   ] LazyLoaded state.apply
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.178.133:4506
[DEBUG   ] Trying to connect to: tcp://192.168.178.133:4506
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.178.133:4506
[DEBUG   ] Trying to connect to: tcp://192.168.178.133:4506
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] In saltenv 'base', looking at rel_path 'testcase/substate.sls' to resolve 'salt://testcase/substate.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/testcase/substate.sls' to resolve 'salt://testcase/substate.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/base/testcase/substate.sls
[DEBUG   ] Jinja search path: [u'/var/cache/salt/minion/files/base']
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.178.133:4506
[DEBUG   ] Trying to connect to: tcp://192.168.178.133:4506
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/testcase/substate.sls' using 'jinja' renderer: 0.0085711479187
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/testcase/substate.sls:
include:
    - .ze-file

[DEBUG   ] Results of YAML rendering:
OrderedDict([(u'include', [u'.ze-file'])])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/testcase/substate.sls' using 'yaml' renderer: 0.000598192214966
[DEBUG   ] Could not find file 'salt://ze-file.sls' in saltenv 'base'
[DEBUG   ] Could not find file 'salt://ze-file/init.sls' in saltenv 'base'
[DEBUG   ] Minion return retry timer set to 5 seconds (randomized)
[INFO    ] Returning information for job: 20190118134335325608
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.178.133:4506
[DEBUG   ] Trying to connect to: tcp://192.168.178.133:4506
[DEBUG   ] minion return: {u'fun_args': [u'testcase/substate'], u'jid': u'20190118134335325608', u'return': [u'Specified SLS ze-file in saltenv base is not available on the salt master or through a configured fileserver'], u'retcode': 1, u'success': True, u'fun': u'state.apply'}

Debug log for working salt $HOSTNAME state.apply testcase.substate

[INFO    ] User sudo_michael Executing command state.apply with jid 20190118134638957362
[DEBUG   ] Command details {u'tgt_type': u'glob', u'jid': u'20190118134638957362', u'tgt': u'salt', u'ret': u'', u'user': u'sudo_michael', u'arg': [u'testcase.substate'], u'fun': u'state.apply'}
[INFO    ] Starting a new job with PID 3317
[DEBUG   ] LazyLoaded state.apply
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.178.133:4506
[DEBUG   ] Trying to connect to: tcp://192.168.178.133:4506
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.178.133:4506
[DEBUG   ] Trying to connect to: tcp://192.168.178.133:4506
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] In saltenv 'base', looking at rel_path 'testcase/substate.sls' to resolve 'salt://testcase/substate.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/testcase/substate.sls' to resolve 'salt://testcase/substate.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/base/testcase/substate.sls
[DEBUG   ] Jinja search path: [u'/var/cache/salt/minion/files/base']
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.178.133:4506
[DEBUG   ] Trying to connect to: tcp://192.168.178.133:4506
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/testcase/substate.sls' using 'jinja' renderer: 0.00863695144653
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/testcase/substate.sls:
include:
    - .ze-file

[DEBUG   ] Results of YAML rendering:
OrderedDict([(u'include', [u'.ze-file'])])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/testcase/substate.sls' using 'yaml' renderer: 0.000686883926392
[DEBUG   ] In saltenv 'base', looking at rel_path 'testcase/ze-file.sls' to resolve 'salt://testcase/ze-file.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/testcase/ze-file.sls' to resolve 'salt://testcase/ze-file.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/base/testcase/ze-file.sls
[DEBUG   ] Jinja search path: [u'/var/cache/salt/minion/files/base']
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.178.133:4506
[DEBUG   ] Trying to connect to: tcp://192.168.178.133:4506
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/testcase/ze-file.sls' using 'jinja' renderer: 0.00725197792053
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/testcase/ze-file.sls:
/tmp/test.dat:
    file.managed:
        - source: salt://testcase/test.dat


[DEBUG   ] Results of YAML rendering:
OrderedDict([(u'/tmp/test.dat', OrderedDict([(u'file.managed', [OrderedDict([(u'source', u'salt://testcase/test.dat')])])]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/testcase/ze-file.sls' using 'yaml' renderer: 0.000854969024658
[DEBUG   ] LazyLoaded config.option
[DEBUG   ] LazyLoaded file.managed
[INFO    ] Running state [/tmp/test.dat] at time 13:46:39.581435
[INFO    ] Executing state file.managed for [/tmp/test.dat]
[DEBUG   ] LazyLoaded file.source_list
[DEBUG   ] LazyLoaded cp.hash_file
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.178.133:4506
[DEBUG   ] Trying to connect to: tcp://192.168.178.133:4506
[DEBUG   ] In saltenv 'base', looking at rel_path 'testcase/test.dat' to resolve 'salt://testcase/test.dat'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/testcase/test.dat' to resolve 'salt://testcase/test.dat'
[INFO    ] File /tmp/test.dat is in the correct state
[INFO    ] Completed state [/tmp/test.dat] at time 13:46:39.599496 (duration_in_ms=18.061)
[DEBUG   ] File /var/cache/salt/minion/accumulator/139899653238160 does not exist, no need to cleanup
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] Minion return retry timer set to 10 seconds (randomized)
[INFO    ] Returning information for job: 20190118134638957362
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'salt', u'tcp://192.168.178.133:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.178.133:4506
[DEBUG   ] Trying to connect to: tcp://192.168.178.133:4506
[DEBUG   ] minion return: {u'fun_args': [u'testcase.substate'], u'jid': u'20190118134638957362', u'return': {u'file_|-/tmp/test.dat_|-/tmp/test.dat_|-managed': {u'comment': u'File /tmp/test.dat is in the correct state', u'pchanges': {}, u'name': u'/tmp/test.dat', u'start_time': '13:46:39.581435', u'result': True, u'duration': 18.061, u'__run_num__': 0, u'__sls__': u'testcase.ze-file', u'changes': {}, u'__id__': u'/tmp/test.dat'}}, u'retcode': 0, u'success': True, u'fun': u'state.apply'}

[DickerDackel]

Ah! I was (for whatever reason) convinced, that the ‘.’ and ‘/’ notation are synonymous, but your explanation from the python point of view makes absolute sense. I’ll change our best practice recommendation and existing wrapper scripts to the dot-notation.

Thank you 😃

[waynew]

TL;DR

Don’t use state.apply some/state/file, use state.apply some.state.file, and everything will be fine!

---

Okay, so this looks like correct, but perhaps unexpected behavior if you’re not familiar with how imports in Python work.

In the docs for state.apply

To apply individual SLS files, pass them as a comma-separated list:

When you’re calling state.apply <sls files> you’re passing in the sls file. When you do this:

salt $HOSTNAME state.apply testcase/ze-file  # <-- works

That works, because you’re passing it an individual sls file. There are no relative includes, so it’s A-OK. When you try this:

salt $HOSTNAME state.apply testcase.substate  # <-- works

The . in the state makes salt include the file as a module, so when looks for relative includes it knows that they should be relative to that module. In your include it becomes testcase.ze-file. On the other hand, when you try:

salt $HOSTNAME state.apply testcase/substate  # <-- fails

It fails, because you’re passing it an individual sls file again, but now that you’re passing it a file name instead of a module name. When it goes to combine all the sls files, it doesn’t know testcase is part of anything! Salt just assumes that whatever is referenced in substate should be included from the root directory (e.g. the default of /srv/salt). So what happens here is that it’s working in the /srv/salt/ directory, sees the include of .ze-file and assumes that should be /srv/salt/ze-file.

You can see this by changing your include to - testcase/ze-file, and you’ll see that it works regardless of how you call state.apply.

I’m not sure if we support the state.apply path/to/statefile (I didn’t find anything in the docs), but path.to.statefile is definitely supported, encouraged, and as you’ve seen with your relative includes, the expected way to do things 🙂