Mitogen fails on `pip` module

• Ansible and Python versions:

ansible --version
ansible 2.7.9
  configured module search path = [u'/home/strangeman/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.16 (default, Apr  6 2019, 01:42:57) [GCC 8.3.0]

• Ansible config:

ANSIBLE_PIPELINING(/mnt/src/sources/playbooks/ansible.cfg) = True
ANSIBLE_SSH_ARGS(/mnt/src/sources/playbooks/ansible.cfg) = -o StrictHostKeyChecking=no -o PreferredAuthentications=publickey -o ControlMaster=auto -o ControlPersist=60s
CACHE_PLUGIN(/mnt/src/sources/playbooks/ansible.cfg) = yaml
CACHE_PLUGIN_CONNECTION(/mnt/src/sources/playbooks/ansible.cfg) = ./ansible-cache
CACHE_PLUGIN_TIMEOUT(/mnt/src/sources/playbooks/ansible.cfg) = 86400
DEFAULT_CALLBACK_WHITELIST(/mnt/src/sources/playbooks/ansible.cfg) = [u'print_changed']
DEFAULT_GATHERING(/mnt/src/sources/playbooks/ansible.cfg) = smart
DEFAULT_LOOKUP_PLUGIN_PATH(/mnt/src/sources/playbooks/ansible.cfg) = [u'/mnt/src/sources/playbooks/lookup_plugins']
DEFAULT_MANAGED_STR(/mnt/src/sources/playbooks/ansible.cfg) = Ansible managed, do not edit directly
DEFAULT_ROLES_PATH(/mnt/src/sources/playbooks/ansible.cfg) = [u'/mnt/src/sources/playbooks/roles']
DEFAULT_STRATEGY(/mnt/src/sources/playbooks/ansible.cfg) = mitogen_linear
DEFAULT_STRATEGY_PLUGIN_PATH(/mnt/src/sources/playbooks/ansible.cfg) = [u'/usr/local/lib/python2.7/dist-packages/ansible_mitogen/plugins/strategy']
DEFAULT_TIMEOUT(/mnt/src/sources/playbooks/ansible.cfg) = 60
RETRY_FILES_ENABLED(/mnt/src/sources/playbooks/ansible.cfg) = False
RETRY_FILES_SAVE_PATH(/mnt/src/sources/playbooks/ansible.cfg) = /mnt/src/sources/playbooks/.ansible-retry

• On target hosts we use Centos 7 with Python 2.7.5
• Mitogen version: 0.2.7 (also tried with master version).
• Sample playbook:

- name: Install gunicorn and wheel
  pip: name={{ item }} executable=/opt/nginxauth/env/bin/pip
  become: true
  become_user: "nginxauth"
  with_items:
    - gunicorn
    - wheel

• Fail log:

The full traceback is:
Traceback (most recent call last):
  File "master:/usr/local/lib/python2.7/dist-packages/ansible_mitogen/runner.py", line 883, in _run
    self._run_code(code, mod)
  File "master:/usr/local/lib/python2.7/dist-packages/ansible_mitogen/runner.py", line 862, in _run_code
    exec('exec code in vars(mod)')
  File "<string>", line 1, in <module>
  File "master:/usr/local/lib/python2.7/dist-packages/ansible/modules/packaging/language/pip.py", line 240, in <module>
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2998, in <module>
    _declare_state('object', working_set = WorkingSet())
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 476, in __init__
    self.add_entry(entry)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 491, in add_entry
    for dist in find_distributions(entry, True):
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1982, in find_on_path
    path_item = _normalize_cached(path_item)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2142, in _normalize_cached
    _cache[filename] = result = normalize_path(filename)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2136, in normalize_path
    return os.path.normcase(os.path.realpath(filename))
  File "/usr/lib64/python2.7/posixpath.py", line 368, in realpath
    return abspath(path)
  File "/usr/lib64/python2.7/posixpath.py", line 356, in abspath
    cwd = os.getcwd()
OSError: [Errno 2] No such file or directory


failed: [testhost.com] (item=[u'gunicorn', u'wheel']) => {
    "changed": false, 
    "item": [
        "gunicorn", 
        "wheel"
    ], 
    "module_stderr": "Traceback (most recent call last):\n  File \"master:/usr/local/lib/python2.7/dist-packages/ansible_mitogen/runner.py\", line 883, in _run\n    self._run_code(code, mod)\n  File \"master:/usr/local/lib/python2.7/dist-packages/ansible_mitogen/runner.py\", line 862, in _run_code\n    exec('exec code in vars(mod)')\n  File \"<string>\", line 1, in <module>\n  File \"master:/usr/local/lib/python2.7/dist-packages/ansible/modules/packaging/language/pip.py\", line 240, in <module>\n  File \"/usr/lib/python2.7/site-packages/pkg_resources.py\", line 2998, in <module>\n    _declare_state('object', working_set = WorkingSet())\n  File \"/usr/lib/python2.7/site-packages/pkg_resources.py\", line 476, in __init__\n    self.add_entry(entry)\n  File \"/usr/lib/python2.7/site-packages/pkg_resources.py\", line 491, in add_entry\n    for dist in find_distributions(entry, True):\n  File \"/usr/lib/python2.7/site-packages/pkg_resources.py\", line 1982, in find_on_path\n    path_item = _normalize_cached(path_item)\n  File \"/usr/lib/python2.7/site-packages/pkg_resources.py\", line 2142, in _normalize_cached\n    _cache[filename] = result = normalize_path(filename)\n  File \"/usr/lib/python2.7/site-packages/pkg_resources.py\", line 2136, in normalize_path\n    return os.path.normcase(os.path.realpath(filename))\n  File \"/usr/lib64/python2.7/posixpath.py\", line 368, in realpath\n    return abspath(path)\n  File \"/usr/lib64/python2.7/posixpath.py\", line 356, in abspath\n    cwd = os.getcwd()\nOSError: [Errno 2] No such file or directory\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", 
    "rc": 1
}

[nigelm]

I think I have just run into the same issue, although it was being disguised for me by me installing python36 from EPEL and using pip with a venv based on python36. However I am also using a target of Centos 7.6.1810 and ansible 2.7.10 (running on a Mac).

In any case, you can work around for this task by setting mitogen_task_isolation: fork - so my previously failing task looks like this:-

- name: build app requirements
  pip:
    requirements: ~healthchecks/webapp/healthchecks/requirements.txt
    virtualenv:   ~healthchecks/webapp/venv
    virtualenv_command: /usr/bin/python3.6 -m venv
  become_user: healthchecks
  vars:
    mitogen_task_isolation: fork

[dw]

This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

Thanks for reporting this!