Error "Got a bad pillar from master, type bool, expecting dict: False"

I got this error on one of my minion and I would like to fix that but I don’t know what is the problematic pillar. The minion is now stuck in a kind of corrupted state. Grains are still working.

I got the exact same roles, pillars and grains on an other gameserver and the error don’t happen on it. I’m on version 0.16.4 for the minion and master. At the same time, is it possible to include the erroneous pillar name in the log.

I tried (on the minion and on the master) :

salt-call saltutil.sync_all
salt-call state.clear_cache
salt-call saltutil.refresh_pillar
service salt-minion restart

Error output:

# salt-call pillar.items
[INFO    ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[ERROR   ] Got a bad pillar from master, type bool, expecting dict: False
[ERROR   ] Got a bad pillar from master, type bool, expecting dict: False
local:
    ----------

Debug output:

# salt-call -l debug pillar.items
[DEBUG   ] Reading configuration from /etc/salt/minion
[INFO    ] Using cached minion ID from /etc/salt/minion_id: 2-qa-gs.xx.local
[DEBUG   ] Configuration file path: /etc/salt/minion
[DEBUG   ] Reading configuration from /etc/salt/minion
[INFO    ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[ERROR   ] Got a bad pillar from master, type bool, expecting dict: False
[DEBUG   ] Loaded localemod as virtual locale
[DEBUG   ] Loaded groupadd as virtual group
[DEBUG   ] Loaded rh_service as virtual service
[DEBUG   ] Loaded yumpkg as virtual pkg
[DEBUG   ] Loaded grub_legacy as virtual grub
[DEBUG   ] Loaded parted as virtual partition
[DEBUG   ] Loaded linux_sysctl as virtual sysctl
[DEBUG   ] Loaded mdadm as virtual raid
[DEBUG   ] Loaded sysmod as virtual sys
[DEBUG   ] Loaded linux_acl as virtual acl
[DEBUG   ] Loaded rpm as virtual lowpkg
[DEBUG   ] Loaded zcbuildout as virtual buildout
[DEBUG   ] Loaded useradd as virtual user
[DEBUG   ] Loaded rh_ip as virtual ip
[DEBUG   ] Loaded virtualenv_mod as virtual virtualenv
[DEBUG   ] Loaded cmdmod as virtual cmd
[DEBUG   ] Loaded linux_lvm as virtual lvm
[DEBUG   ] Loaded djangomod as virtual django
[DEBUG   ] Loaded couchdb_return as virtual couchdb
[DEBUG   ] Loaded syslog_return as virtual syslog
[DEBUG   ] Loaded carbon_return as virtual carbon
[DEBUG   ] Loaded sqlite3_return as virtual sqlite3
[DEBUG   ] Loaded pip_state as virtual pip
[DEBUG   ] Loaded saltmod as virtual salt
[DEBUG   ] Loaded mdadm as virtual raid
[DEBUG   ] Loaded zcbuildout as virtual buildout
[DEBUG   ] Loaded virtualenv_mod as virtual virtualenv
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[ERROR   ] Got a bad pillar from master, type bool, expecting dict: False
[DEBUG   ] Loaded no_out as virtual quiet
[DEBUG   ] Loaded json_out as virtual json
[DEBUG   ] Loaded yaml_out as virtual yaml
[DEBUG   ] Loaded pprint_out as virtual pprint

[kcgen]

Update to my previous message:

The problem appears related to the new pillar_cache feature in the 2015.8.8 master when using 2015.8.7 minions. I had that enabled pillar_cache on the master (in /etc/salt/master), however it seems the master’s config values become pillars in the minion, which it didn’t like.

Here’s the error message from the master:

2016-03-28 12:27:54,308 [salt.transport.mixins.auth                           ][INFO    ][6948] Authentication request from p-run2
2016-03-28 12:27:54,309 [salt.transport.mixins.auth                           ][INFO    ][6948] Authentication accepted from p-run2
2016-03-28 12:27:54,712 [salt.pillar                                          ][INFO    ][6957] Determining pillar cache
2016-03-28 12:27:54,712 [salt.pillar                                          ][INFO    ][6957] Compiling pillar from cache
2016-03-28 12:27:54,713 [salt.utils.cache                                     ][INFO    ][6957] Factory backend: memory
2016-03-28 12:27:54,713 [salt.master                                          ][ERROR   ][6957] Error in function _pillar:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/salt/master.py", line 1411, in run_func
    ret = getattr(self, func)(load)
  File "/usr/lib/python2.6/site-packages/salt/master.py", line 1147, in _pillar
    data = pillar.compile_pillar(pillar_dirs=pillar_dirs)
  File "/usr/lib/python2.6/site-packages/salt/pillar/__init__.py", line 248, in compile_pillar
    log.debug('Scanning cache: {0}'.format(self.cache._dict))
AttributeError: 'CacheDict' object has no attribute '_dict'

Disabling the pillar_cache on the master (bysetting it to False instead of True) allowed the minions to start up without issue.

[kaidokert]

I just ran into the same thing, and doing

pillar_cache: False

in /etc/salt/master.d/blah.conf indeed works around it.