Mitogen fails on RHEL8 server with bash: /usr/bin/python: No such file or directory

Incorrect identification of python interpreter on Red Hat Enterprise Linux 8.0. Cause playbook to fail instantly.

Hope this issue is easy to find and solve, this is a awesome plugin that speeds up our Ansible playbooks with 9x, absolutely mind blowing, can’t live without it. Great work!

fatal: [rhel8-test]: UNREACHABLE! => changed=false
  msg: |-
    EOF on stream; last 100 lines received:
    bash: /usr/bin/python: No such file or directory
  unreachable: true

• Which version of Ansible are you running? 2.8.5

ansible 2.8.5
  config file = /opt/tools/rlinq/dev/ansible/core/ansible.cfg
  configured module search path = [u'/export/home/rlinq/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

• Is your version of Ansible patched in any way?

No

• Are you running with any custom modules, or module_utils loaded?

No

• Have you tried the latest master version from Git?

No, but latest released version mitogen-0.2.9 is used in this issue report. Reviewed diff between 0.2.9 and master found nothing related to python version detection.

• How to reproduce the issue

Install a RHEL8 or possible a CENTOS8 server (haven’t been able to test on CENTOS8 but should be the same) Download mitogen 0.2.9 and extract that in a path of your choice. Configure this path in your ansible.cfg file in strategy_plugin. Set strategy = mitogen_linear

rlinq@rhel8-test $ grep mitogen ../ansible.cfg
strategy_plugins = /opt/tools/rlinq/tools/mitogen-0.2.9/ansible_mitogen/plugins/strategy
strategy = mitogen_linear

Create a simple ansible playbook that connects to your self using hostname.

rlinq@rhel8-test $ cat mitogen_test_rhel8.yml
- hosts: rhel8-test
  gather_facts: 'no'
  strategy: mitogen_linear
  pre_tasks:
    - setup:
        gather_subset: 'network'

rlinq@rhel8-test $ ansible-playbook mitogen_test_rhel8.yml -i hosts -vvv'

Result

rlinqmgr@rhel8-test: /.../rlinq_ansible/roles $ ansible-playbook mitogen_test_rhel8.yml -i ../environments/test/hosts -vvv
ansible-playbook 2.9.1
  config file = /opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg
  configured module search path = ['/export/home/rlinqmgr/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Jun 12 2019, 01:12:31) [GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]
Using /opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg as config file
host_list declined parsing /opt/tools/rlinq/clones/rlinq_ansible/environments/test/hosts as it did not pass its verify_file() method
script declined parsing /opt/tools/rlinq/clones/rlinq_ansible/environments/test/hosts as it did not pass its verify_file() method
auto declined parsing /opt/tools/rlinq/clones/rlinq_ansible/environments/test/hosts as it did not pass its verify_file() method
Parsed /opt/tools/rlinq/clones/rlinq_ansible/environments/test/hosts inventory source with ini plugin

PLAYBOOK: mitogen_test_rhel8.yml ****************************************************************************************************************************
1 plays in mitogen_test_rhel8.yml

PLAY [rhel8-test] **************************************************************************************************************************************
[top  18273] 16:11:06.072438 D ansible_mitogen.affinity: CPU mask for Ansible top-level process: 0x000002
[top  18273] 16:11:06.072951 D ansible_mitogen.process: inherited open file limits: soft=1024 hard=4096
[top  18273] 16:11:06.073176 D ansible_mitogen.process: raised soft open file limit from 1024 to 4096
[mux  18280] 16:11:06.076797 D ansible_mitogen.affinity: CPU mask for MuxProcess 0: 0x000001
[mux  18280] 16:11:06.086035 D mitogen.service: Pool(9828, size=32, th='MainThread'): initialized
[mux  18280] 16:11:06.087603 D ansible_mitogen.process: Service pool configured: size=32

TASK [setup] ************************************************************************************************************************************************
task path: /opt/tools/rlinq/clones/rlinq_ansible/roles/mitogen_test_rhel8.yml:5
[task 18314] 16:11:06.151910 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000002
[task 18314] 16:11:06.168864 D ansible_mitogen.process: will use multiplexer 0 (/tmp/mitogen_unix_acgc811n.sock) to connect to "rhel8-test"
[task 18314] 16:11:06.169072 D mitogen.unix: client: connecting to /tmp/mitogen_unix_acgc811n.sock
[task 18314] 16:11:06.169753 D mitogen.unix: client: local ID is 1, remote is 0
[mux  18280] 16:11:06.169885 D mitogen.unix: listener: accepted connection from PID 18314: unix_client.18314
[mux  18280] 16:11:06.173997 D mitogen.parent: creating connection to context 2 using mitogen.ssh
[mux  18280] 16:11:06.350272 D mitogen.parent: command line for Connection(None): ssh -o "LogLevel ERROR" -i /export/home/rlinqmgr/secret/rlinqmgr_secret_id_rsa -o "Compression yes" -o "ServerAliveInterval 30" -o "ServerAliveCountMax 10" -o "BatchMode yes" -o "StrictHostKeyChecking yes" -F ./ssh_config rhel8-test /usr/bin/python -c "'import codecs,os,sys;_=codecs.decode;exec(_(_(REMOVED_ON_PURPOSE=\".encode(),\"base64\"),\"zip\"))'"
[mux  18280] 16:11:06.357355 D mitogen.parent: child for Connection(None) started: pid:18317 stdin:76 stdout:76 stderr:78
[mux  18280] 16:11:06.869922 D mitogen.parent: ssh.rhel8-test: (unrecognized): bash: /usr/bin/python: No such file or directory
[mux  18280] 16:11:06.870307 D mitogen: <Side of ssh.rhel8-test fd 76>: empty read, disconnecting
[mux  18280] 16:11:06.871088 D mitogen: BootstrapProtocol(ssh.rhel8-test): disconnecting
[mux  18280] 16:11:06.871509 D mitogen: <Side of ssh.rhel8-test fd 78>: empty read, disconnecting
[mux  18280] 16:11:06.871673 D mitogen.parent: failing connection ssh.rhel8-test due to EofError('EOF on stream; last 100 lines received:\nbash: /usr/bin/python: No such file or directory',)
[mux  18280] 16:11:06.872036 D mitogen.parent: PopenProcess ssh.rhel8-test pid 18317: exited with return code 127
[mux  18280] 16:11:06.872191 D mitogen: SetupProtocol(ssh.rhel8-test): disconnecting
[mux  18280] 16:11:06.872615 D mitogen: SetupProtocol(ssh.rhel8-test): disconnecting
[task 18314] 16:11:06.884586 D mitogen: MitogenProtocol(unix_listener.18280): disconnecting
[task 18314] 16:11:06.885241 D mitogen: Waker(fd=10/11): disconnecting
[task 18314] 16:11:06.885705 D mitogen: Router(Broker(37f0)): stats: 0 module requests in 0 ms, 0 sent (0 ms minify time), 0 negative responses. Sent 0.0 kb total, 0.0 kb avg.
[mux  18280] 16:11:06.885811 D mitogen: <Side of unix_client.18314 fd 75>: empty read, disconnecting
[mux  18280] 16:11:06.885993 D mitogen: MitogenProtocol(unix_client.18314): disconnecting
fatal: [rhel8-test]: UNREACHABLE! => changed=false
  msg: |-
    EOF on stream; last 100 lines received:
    bash: /usr/bin/python: No such file or directory
  unreachable: true

PLAY RECAP **************************************************************************************************************************************************
rhel8-test            : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

Running against localhost, this results in an OK identification of python3 version

Included only snippet until python path is identified and connection is setup.

TASK [Gathering Facts] **************************************************************************************************************************************
task path: /opt/tools/rlinq/clones/rlinq_ansible/roles/mitogen_test_rhel8.yml:1
[task 30354] 22:05:54.515629 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000002
[task 30354] 22:05:54.526037 D ansible_mitogen.process: will use multiplexer 0 (/tmp/mitogen_unix_n73udz9n.sock) to connect to "localhost"
[task 30354] 22:05:54.526285 D mitogen.unix: client: connecting to /tmp/mitogen_unix_n73udz9n.sock
[task 30354] 22:05:54.527010 D mitogen.unix: client: local ID is 1, remote is 0
[mux  30320] 22:05:54.527165 D mitogen.unix: listener: accepted connection from PID 30354: unix_client.30354
[mux  30320] 22:05:54.528993 D mitogen.parent: creating connection to context 2 using mitogen.parent
[mux  30320] 22:05:54.711130 D mitogen.parent: command line for Connection(None): /usr/bin/python3 -c "import codecs,os,sys;_=codecs.decode;exec(_(_(\"eNqFkU9Lw0AQxc/Np8htd+mablLEElhQehAPIgSxBy2SP5t2abq77G4a66d3TMUm9eBtfsybeY+ZjK64dpGRRmASWNoNSNYhQK3tDpM0mEBdtSbBjMaMkTNndEgWuvGJy0Y7gbMh2CGshtABgKE7gn2Te3Ddh5yHqMptJxUKc1X1TfEhytbnRSP69qx1dlZINTNHv9UKQc7JhWzK+8GDsE5q9ZrO172tUAdpgdFddv/C0JqPx04awAaPG3SMU4T30uuNUKkzTat2+429le7KbkWzKFvn2bWLtBJ5ZSMlfDpn84QRRAJY31npBY4penx4fmKMvSkEsUpdwfVJsOTv+Pv+lTZCwdWRLRCJLGzC8Q1LFoSiT2lgU234WbeiqCvQ90tq82Ow7OvTmS/U3X/qvynjccrfBybkC0uYucA=\".encode(),\"base64\"),\"zip\"))"
[mux  30320] 22:05:54.718144 D mitogen.parent: child for Connection(None) started: pid:30357 stdin:76 stdout:76 stderr:None
[mux  30320] 22:05:54.759414 D mitogen.parent: BootstrapProtocol(local.30357): first stage started succcessfully
[mux  30320] 22:05:54.764670 D mitogen.parent: BootstrapProtocol(local.30357): first stage received mitogen.core source
[mux  30320] 22:05:54.833924 D mitogen.parent: BootstrapProtocol(local.30357): new child booted successfully
[mux  30320] 22:05:54.834463 D mitogen.[local.30357]: Python version is 3.6.8 (default, Jun 12 2019, 01:12:31)
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]
[mux  30320] 22:05:54.834621 D mitogen.[local.30357]: Parent is context 0 (master); my ID is 2
[mux  30320] 22:05:54.834739 D mitogen.[local.30357]: pid:30357 ppid:30320 uid:38001942/38001942, gid:38002279/38002279 host:'rhel8-test.company.com'
[mux  30320] 22:05:54.834848 D mitogen.[local.30357]: Recovered sys.executable: '/usr/bin/python3'
[mux  30320] 22:05:54.835892 D mitogen.parent: starting function call to local.30357: ansible_mitogen.target.init_child(log_level=10, candidate_temp_dirs=['/tmp', '/var/tmp', '/tmp'])

• Do you have some idea of what the underlying problem may be?

Redhat 8 Server doesn’t have /usr/bin/python, the mitogen code handles remote connection different from localhost. Where localhost successfully identifies correct python path a remote connection to the same host fails.

• Mention your host and target OS and versions Same behavior seen using RHEL7 or RHEL8 as host towards a RHEL8 target. Logs only include from RHEL8 -> RHEL8. Tests conducted from RHEL7-RHEL8 with the same result.

• Host1

  • Red Hat Enterprise Linux release 8.0 (Oopta)
  • Linux rhel8-test 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Sun Sep 15 11:24:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Host2

  • Red Hat Enterprise Linux Server release 7.7 (Maipo)
  • Linux rhel7-test 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 16 14:19:51 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

• Target

  • Red Hat Enterprise Linux release 8.0 (Oopta)
  • Linux rhel8-test 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Sun Sep 15 11:24:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Mention your host and target Python versions

• Host1

  • Python 3.6.8

• Host2

  • Python 2.7.5

• Target

  • Python 3.6.8

• If reporting a performance issue, mention the number of targets and a rough description of your workload (lots of copies, lots of tiny file edits, etc.)

• If reporting a crash or hang in Ansible, please rerun with -vvv and include 200 lines of output around the point of the error, along with a full copy of any traceback or error text in the log. Beware “-vvv” may include secret data! Edit as necessary before posting.

• If reporting any kind of problem with Ansible, please include the Ansible version along with output of “ansible-config dump --only-changed”.

ANSIBLE_SSH_ARGS(/opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg) = -F ./ssh_config
DEFAULT_CALLBACK_PLUGIN_PATH(/opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg) = ['/usr/share/ansible/plugins/callback', '/opt/tools/rlinq/dev/a>
DEFAULT_FORKS(/opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg) = 100
DEFAULT_HOST_LIST(/opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg) = ['/opt/tools/rlinq/clones/rlinq_ansible/roles/environments/test']
DEFAULT_LOAD_CALLBACK_PLUGINS(/opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg) = True
DEFAULT_PRIVATE_KEY_FILE(/opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg) = /export/home/rlinqmgr/secret/rlinqmgr_secret_id_rsa
DEFAULT_STDOUT_CALLBACK(/opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg) = yaml
DEFAULT_STRATEGY(/opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg) = mitogen_linear
DEFAULT_STRATEGY_PLUGIN_PATH(/opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg) = ['/opt/tools/rlinq/tools/mitogen-0.2.9/ansible_mitogen/plugins/>
DEFAULT_TIMEOUT(/opt/tools/rlinq/clones/rlinq_ansible/roles/ansible.cfg) = 60

[s1113950]

New rc available here: https://github.com/dw/mitogen/releases/tag/v0.3.0-rc.0 for Ansible 2.10+ For Ansible < 2.10, there’s also https://github.com/dw/mitogen/releases/tag/v0.2.10-rc.0 both have the fix in them

[s1113950]

@le0m I don’t have the ability to push to Mitogen’s PyPi account, so that would be up to @dw . There are already plans to cut a release after Ansible 2.10 comes out though, which is expected on August 13th.