Proxy settings don’t work for new pack install method

Pls refer to issue: https://github.com/StackStorm/st2docs/issues/352

“st2 pack install openstack” (openstack pack taken as example), does not work event after giving “git config --global http.proxy http://user:[e-mail]:port” as suggested in the documentation. If “git clone” for the repo is done manually on the command prompt it works, but fails via “packs.download” action.

“st2 search” works with modifications suggested in Modify: /opt/stackstorm/st2/lib/python2.7/site-packages/st2common/services/packs.py (st2docs/issues/352)

brocade@BWC2:~$ st2 pack search openstack
+-----------+--------------------------------------------+---------+------------------+
| name      | description                                | version | author           |
+-----------+--------------------------------------------+---------+------------------+
| libcloud  | st2 content pack containing libcloud       | 0.1.0   | StackStorm, Inc. |
|           | integrations                               |         |                  |
| openstack | OpenStack integration pack                 | 0.1.0   | st2dev           |
+-----------+--------------------------------------------+---------+------------------+

But pack install for “openstack” pack fails with error " unable to access ‘https://github.com/StackStorm-Exchange/stackstorm-openstack/’: Could not resolve host: github.com".

brocade@BWC2:~$ st2 pack install openstack

        [  failed   ] download pack

id: 585a450ae79fa05ed986edc9
action.ref: packs.install
parameters:
  packs:
  - openstack
status: failed
error: Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 242, in <module>
    obj.run()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 154, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 75, in run
    verifyssl=verifyssl, ref=pack_version)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 102, in _clone_repo
    repo = Repo.clone_from(repo_url, temp_dir)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/repo/base.py", line 925, in clone_from
    return cls._clone(git, url, to_path, GitCmdObjectDB, progress, **kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/repo/base.py", line 880, in _clone
    finalize_process(proc, stderr=stderr)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/util.py", line 336, in finalize_process
    proc.wait(**kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/cmd.py", line 290, in wait
    raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v https://github.com/StackStorm-Exchange/stackstorm-openstack /root/bd373ecc09ed3463a0b7849cc13d7dc4
  stderr: 'Cloning into '/root/bd373ecc09ed3463a0b7849cc13d7dc4'...
fatal: unable to access 'https://github.com/StackStorm-Exchange/stackstorm-openstack/': Could not resolve host: github.com

After adding proxy details in the git config, able to to manually go a “git clone”:

brocade@BWC2:~$ git config --global http.proxy http://10.254.2.5:3128
brocade@BWC2:~$ git clone https://github.com/StackStorm-Exchange/stackstorm-openstack
Cloning into 'stackstorm-openstack'...
remote: Counting objects: 823, done.
remote: Total 823 (delta 0), reused 0 (delta 0), pack-reused 823
Receiving objects: 100% (823/823), 136.74 KiB | 0 bytes/s, done.
Resolving deltas: 100% (649/649), done.
Checking connectivity... done.

But “st2 pack install…” still fails:

brocade@BWC2:~$ st2 pack install openstack

        [  failed   ] download pack

id: 585a46b4e79fa05ed986edcc
action.ref: packs.install
parameters:
  packs:
  - openstack
status: failed
error: Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 242, in <module>
    obj.run()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 154, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 75, in run
    verifyssl=verifyssl, ref=pack_version)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 102, in _clone_repo
    repo = Repo.clone_from(repo_url, temp_dir)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/repo/base.py", line 925, in clone_from
    return cls._clone(git, url, to_path, GitCmdObjectDB, progress, **kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/repo/base.py", line 880, in _clone
    finalize_process(proc, stderr=stderr)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/util.py", line 336, in finalize_process
    proc.wait(**kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/cmd.py", line 290, in wait
    raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v https://github.com/StackStorm-Exchange/stackstorm-openstack /root/bd373ecc09ed3463a0b7849cc13d7dc4
  stderr: 'Cloning into '/root/bd373ecc09ed3463a0b7849cc13d7dc4'...
fatal: unable to access 'https://github.com/StackStorm-Exchange/stackstorm-openstack/': Could not resolve host: github.com
'

traceback: None
failed_on: download pack
start_timestamp: 2016-12-21T09:09:08.163215Z
end_timestamp: 2016-12-21T09:09:35.743867Z
+--------------------------+----------------------+---------------+----------------+-------------------------------+
| id                       | status               | task          | action         | start_timestamp               |
+--------------------------+----------------------+---------------+----------------+-------------------------------+
| 585a46b4e79fa05de3e3ccea | failed (27s elapsed) | download pack | packs.download | Wed, 21 Dec 2016 09:09:08 UTC |
+--------------------------+----------------------+---------------+----------------+-------------------------------+

[ClarksonCJ]

@martypare glad I could help, I am working on a PR to issue to @armab et al. with a more generic fix so hopefully in the near future there will be no need to hack the virtualenv file like this 😉

[ClarksonCJ]

I had an additional issue after I had used the workaround suggested by @LindsayHill. Where pip cannot download the dependancies for a virtual environment. I received this error “Exception: Failed to install requirement “six>=1.9.0,<2.0”: Collecting six<2.0,>=1.9.0”, and also noticed there was no log in my CNTLM trace. I had to modify the st2common/util/virtualenvs.py file and add the proxy environment variables to the get_env_for_subprocess_command() method e.g. env['HTTP_PROXY'] = "<proxy_url>" env['HTTPS_PROXY'] = "<proxy_url>"