Docker containers get incorrect HTTP_PROXY injected if the host has HTTP_PROXY=http://127.0.0.1:xxx/

Describe the bug Running the runner in an environment such that the HTTP proxy runs on the same host as the runner, and the HTTP_PROXY et all environment variables are set to http://127.0.0.1:xxx/, results in docker containers being configured to have their HTTP_PROXY et all environment variables set to http://127.0.0.1:xxx/. But docker containers by default run in the default bridge network, so connecting to 127.0.0.1:xxx from container basically always results in connection refused errors.

You could argue running the HTTP proxy on the same host is a dumb idea, because it means that docker containers cannot access the HTTP proxy, thereby breaking any docker actions that need to use the proxy. But in my environment of self-hosted runners we do the following: run a local HTTP forward proxy (that forwards to a parent proxy) as a docker container with a port published to the host, and set HTTP_PROXY et all environment variables of any other docker containers (on the default bridge network) by configuring $HOME/.docker/config.json (see docs). This way we can centrally configure credentials for the parent proxy and we can centrally configure a proxy bypass list. Without the central HTTP forward proxy we would require that any possible tools running in jobs:

1. Support the authentication scheme to the parent proxy.
2. Consistently interpret the proxy bypass list (but many tools don’t support CIDR notation, for example). Since our local HTTP forward proxy is written in Go we support CIDR notation in the proxy bypass list.

Running the proxy not on the same host, but on another compute unit, is not an ideal alternative because of 1 (authentication to the proxy).

It seems that docker container proxy configuration configured in $HOME/.docker/config.json (see docs) is always overridden.

Recently many workflows on self-hosted runners broke because they relied on pre https://github.com/actions/runner/pull/840 behavior.

To Reproduce Steps to reproduce the behavior:

1. Run a runner with a HTTP proxy set to http://127.0.0.1:xxx.
2. Run a step with uses: docker:// syntax that needs to use the proxy.
3. Observe that the container of the step gets a connection refused error.

Expected behavior If the runner’s HTTP_PROXY et all environment variables are set to localhost or 127.0.0.1 (and/or if $HOME/.docker/config.json, docs is configured then docker containers do not explicitly get HTTP_PROXY set.

Runner Version and Platform

Version of your runner? https://github.com/actions/runner/releases/tag/v2.275.1

OS of the machine running the runner? OSX/Windows/Linux/… Linux

What’s not working?

A step with uses: docker:// such that the container does an HTTP request via the HTTP proxy configured in HTTP_PROXY gets a connection refused error because HTTP_PROXY is set to 127.0.0.1.

[richard-bain]

We have a work around for this issue. The following assumes your runners are linux based and your proxy is running in a container:

1. Find the gateway IP for your host: docker network inspect bridge -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}')
2. Use that in your http/s proxy env variables on the host
3. Ensure this gateway ip is in no_proxy

This works because from the host’s perspective this is a loopback to itself e.g. 172.17.0.1:3128 and a container on both the default bridge network and also any user-defined bridge network can also access this address.

Going one step further. if you are able to agree to a convention with your users you could do something like add no_proxy=local,... and remove the .local domain from the proxy. If everyone is prepared to do something like docker run ... --network-alias=<name>.local then this no proxy would also allow name based user-defined networking to work without having to modify the proxy every time.

FWIW I raised an upstream issue on libnetwork but don’t really expect action.

[brekelj1]

Closing due to the elegant workaround discovered by @richard-bain