[BUG] 06.11.2020 -- salt-ssh -- salt doesn't properly recognize appropriate python versions

Description

I have a Centos 7 master and an Centos 7 client. salt-ssh exits with the following error:

ERROR: Python version error. Recommendation(s) follow:
- Install Python 2.6/2.7 Salt dependencies on the Salt SSH 
  master to interact with Python 2.6/2.7 targets
- Install Python 3 on the target machine(s)

Centos 7 Client has python versions: 2.6 & 2.7.5.

Centos 7 Host, where I run salt-ssh from, has python 2.6, 2.7.5, and python 3.6.8.

I believe this may be a bug as I recently ran this from a manjaro box, identical setup, and there were 0 issues. I had to rekick the manjaro box to centos over the weekend, arrived at this issue, and have been unable to find a solution.

Setup

Here is the roster file:

<REDACTED>:
  host: <REDACTED>
  user: <REDACTED>
  passwd: <REDACTED>
  sudo_user: <REDACTED>
  thin_dir: /home/<REDACTED>/.salt_auto

Steps to Reproduce the behavior Here is what I ran, followed by the trace log output. It is confusing to me, as it looks like it searches and eventually finds python2.7, and successfully runs, then returns that it couldn’t find it. Uncertain if it is relevant, but running with -r for raw ssh commands will succeed without issues. The error mentioned occurs regardless of whether I try to apply states, or run other execution modules.

[...]
ERROR: Python version error. Recommendation(s) follow:
- Install Python 2.6/2.7 Salt dependencies on the Salt SSH 
  master to interact with Python 2.6/2.7 targets
- Install Python 3 on the target machine(s)

--------------------------------------------------------------------------------
echo "ERROR: Unable to locate appropriate python command" >&2
exit $EX_PYTHON_INVALID
EOF
[TRACE   ] <REDACTED>'s password:
[TRACE   ] + SUDO=
+ '[' -n '' ']'
+ SUDO_USER=<REDACTED>
+ '[' '' ']'
+ '[' '' ']'
+ EX_PYTHON_INVALID=10
+ PYTHON_CMDS='python3 python27 python2.7 python26 python2.6 python2 python'
[TRACE   ] + for py_cmd in '$PYTHON_CMDS'
+ command -v python3
+ continue
+ for py_cmd in '$PYTHON_CMDS'
+ command -v python27
+ continue
+ for py_cmd in '$PYTHON_CMDS'
+ command -v python2.7
+ python2.7 -c 'import sys; sys.exit(not (sys.version_info >= (2, 6)));'
[TRACE   ] ++ python2.7 -c 'from __future__ import print_function;import sys; print(sys.executable);'
[TRACE   ] + py_cmd_path=/usr/bin/python2.7
++ command -v python2.7
[TRACE   ] + cmdpath=/usr/bin/python2.7
+ grep 'shell script'
+ file /usr/bin/python2.7
[TRACE   ] + exec /usr/bin/python2.7 -c 'import base64;
[...]
[TRACE   ] "").decode("utf-8"))'
[DEBUG   ] End of file(EOL). Brain-dead platform.
[TRACE   ] STDOUT <REDACTED>
<REDACTED>'s password: 
/usr/bin/scp

[TRACE   ] STDERR <REDACTED>
+ SUDO=
+ '[' -n '' ']'
+ SUDO_USER=<REDACTED>
+ '[' '' ']'
+ '[' '' ']'
+ EX_PYTHON_INVALID=10
+ PYTHON_CMDS='python3 python27 python2.7 python26 python2.6 python2 python'
+ for py_cmd in '$PYTHON_CMDS'
+ command -v python3
+ continue
+ for py_cmd in '$PYTHON_CMDS'
+ command -v python27
+ continue
+ for py_cmd in '$PYTHON_CMDS'
+ command -v python2.7
+ python2.7 -c 'import sys; sys.exit(not (sys.version_info >= (2, 6)));'
++ python2.7 -c 'from __future__ import print_function;import sys; print(sys.executable);'
+ py_cmd_path=/usr/bin/python2.7
++ command -v python2.7
+ cmdpath=/usr/bin/python2.7
+ grep 'shell script'
+ file /usr/bin/python2.7
+ exec /usr/bin/python2.7 -c 'import base64;
[...]
[DEBUG   ] RETCODE <REDACTED>: 10
[DEBUG   ] LazyLoaded nested.output
[TRACE   ] data = {'<REDACTED>': {'stdout': 'ERROR: Python version error. Recommendation(s) follow:\n- Install Python 2.6/2.7 Sa
lt dependencies on the...

If I remove the thin directory on the client, and run without using raw ssh commands, it returns there was an issue dpeloying the thin, however on the client, it looks like it deployed successfully.

Master:

[~]$ time salt-ssh -c ~/.salt '*' 'uptime'
[ERROR   ] ERROR: Failure deploying thin, retrying:
STDOUT:
<REACTED>'s password: 

STDERR:

RETCODE: 10
<REACTED>:
    ----------
    retcode:
        10
    stderr:
    stdout:
        ERROR: Python version error. Recommendation(s) follow:
        - Install Python 2.6/2.7 Salt dependencies on the Salt SSH 
          master to interact with Python 2.6/2.7 targets
        - Install Python 3 on the target machine(s)

Client during and after install of thin:

[~]# ls /home/<REACTED>/.salt_auto/
salt-thin.tgz
[~]# ls /home/<REACTED>/.salt_auto/
code-checksum  minion  py3  pyall  salt-call  supported-versions  version

Expected behavior salt-ssh runs execution modules, or anything really that isn’t raw ssh, without an error.

Versions Report

[~]$ salt --versions-report

Salt Version:
           Salt: 3000.3
 
Dependency Versions:
           cffi: 1.14.0
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.2
        libgit2: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.20
       pycrypto: 2.6.1
   pycryptodome: 3.9.7
         pygit2: Not Installed
         Python: 3.6.8 (default, Apr  2 2020, 13:34:55)
   python-gnupg: Not Installed
         PyYAML: 5.3.1
          PyZMQ: 15.3.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.8.2003 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1127.10.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.8.2003 Core

From the thin client:

[ ~]# cat /home/<REACTED>/.salt_auto/version 
3000.3

Additional context

I am brand new to salt! Started learning a few weeks ago, apologies if I missed something simple, or haven’t got all the lingo down yet.

[Ch3LL]

I’m going to close this issue then, because yes it would have been great if the thin was generated after the python2 dependencies were installed correctly, but my PR removes this part of the code since python 2 is no longer supported. As you stated @isbm things can be improved here in general though and we are currently working on heist (https://gitlab.com/saltstack/pop/heist) which does improve on some of the flaws of Salt-SSH and runs a full salt minion and does not use the thin concept anymore.

[isbm]

@Ch3LL well, that’s what you’ve just confirmed in different words: “after they were installed correctly”. That is, if ecosystem of Salt was changed, thin should be immediately obsoleted and regenerated. I do agree it is not that easy due to Salt SSH design choices. I still not sure if current design makes any sense in terms of keeping stuff in /tmp/ rather then install it in /opt permanently into a self-contained environment and have a passive client there instead. So you could do a simple SSH call and invoke fully done client, rather then sending limited thin back and forth. Salt SSH is anyway not client-less technically, it just feels so. Thus from this perspective taking disk space in /tmp is the same as taking the same disk space in /opt, basically. Because the outcome is anyway identical.