[BUG] salt-master not running, unable to read key master.pem

Description Salt-master refused to start after reboot, citing Unable to read key: /etc/salt/pki/master/master.pem; passphrase may be incorrect as the issue

Setup Running on Ubuntu 16.04 that has been updated from 14.04 quite a while ago… Upstart is installed along systemd, service as been running fine for a few years like this.

Steps to Reproduce the behavior salt-master debug:

salt-master -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/nodegroups.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/nodegroups.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] Setting up the Salt Master
[DEBUG   ] Marking 'base64_encode' as a jinja filter
[DEBUG   ] Marking 'base64_decode' as a jinja filter
[DEBUG   ] Marking 'md5' as a jinja filter
[DEBUG   ] Marking 'sha1' as a jinja filter
[DEBUG   ] Marking 'sha256' as a jinja filter
[DEBUG   ] Marking 'sha512' as a jinja filter
[DEBUG   ] Marking 'hmac' as a jinja filter
[DEBUG   ] Marking 'hmac_compute' as a jinja filter
[DEBUG   ] Marking 'random_hash' as a jinja filter
[DEBUG   ] Marking 'rand_str' as a jinja filter
[DEBUG   ] Marking 'file_hashsum' as a jinja filter
[DEBUG   ] Marking 'http_query' as a jinja filter
[DEBUG   ] Marking 'strftime' as a jinja filter
[DEBUG   ] Marking 'date_format' as a jinja filter
[DEBUG   ] Marking 'raise' as a jinja global
[DEBUG   ] Marking 'match' as a jinja test
[DEBUG   ] Marking 'equalto' as a jinja test
[DEBUG   ] Marking 'skip' as a jinja filter
[DEBUG   ] Marking 'sequence' as a jinja filter
[DEBUG   ] Marking 'to_bool' as a jinja filter
[DEBUG   ] Marking 'indent' as a jinja filter
[DEBUG   ] Marking 'tojson' as a jinja filter
[DEBUG   ] Marking 'quote' as a jinja filter
[DEBUG   ] Marking 'regex_escape' as a jinja filter
[DEBUG   ] Marking 'regex_search' as a jinja filter
[DEBUG   ] Marking 'regex_match' as a jinja filter
[DEBUG   ] Marking 'regex_replace' as a jinja filter
[DEBUG   ] Marking 'uuid' as a jinja filter
[DEBUG   ] Marking 'unique' as a jinja filter
[DEBUG   ] Marking 'min' as a jinja filter
[DEBUG   ] Marking 'max' as a jinja filter
[DEBUG   ] Marking 'avg' as a jinja filter
[DEBUG   ] Marking 'union' as a jinja filter
[DEBUG   ] Marking 'intersect' as a jinja filter
[DEBUG   ] Marking 'difference' as a jinja filter
[DEBUG   ] Marking 'symmetric_difference' as a jinja filter
[DEBUG   ] Marking 'method_call' as a jinja filter
[DEBUG   ] Marking 'yaml_dquote' as a jinja filter
[DEBUG   ] Marking 'yaml_squote' as a jinja filter
[DEBUG   ] Marking 'yaml_encode' as a jinja filter
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] salt.crypt._get_key_with_evict: Loading private key
[ERROR   ] Unable to read key: /etc/salt/pki/master/master.pem; passphrase may be incorrect
Unable to read key: /etc/salt/pki/master/master.pem; passphrase may be incorrect

Expected behavior That it works 😕

Screenshots If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 3001
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
         Jinja2: 2.8
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.7
         pygit2: Not Installed
         Python: 3.5.2 (default, Apr 16 2020, 17:47:17)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 17.1.2
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4
 
System Versions:
           dist: ubuntu 16.04 Xenial Xerus
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-184-generic
         system: Linux
        version: Ubuntu 16.04 Xenial Xerus

Additional context Add any other context about the problem here.

[mgenti]

Thanks to @baby-gnu that also worked for me. Installed M2Crypto 0.38.0 and then upgraded to the latest salt

[baby-gnu]

Hello.

This is only for the record since I hit the same issue.

My problem comes from pycrptodome consistency check:

>>> from Cryptodome.PublicKey import RSA
>>> path = '/etc/salt/pki/master/master.pem'
>>> key_fh = open(path)
>>> RSA.importKey(key_fh.read())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/Cryptodome/PublicKey/RSA.py", line 727, in import_key
    return _import_keyDER(der, passphrase)
  File "/usr/lib/python3/dist-packages/Cryptodome/PublicKey/RSA.py", line 670, in _import_keyDER
    raise ValueError("RSA key format is not supported")
ValueError: RSA key format is not supported

I bypass this issue by installing the preferred M2Crypto library which does not trigger this issue:

>>> from M2Crypto import RSA
>>> path = '/etc/salt/pki/master/master.pem'
>>> RSA.load_key(path)
<M2Crypto.RSA.RSA object at 0x7fd2a03f3c88>