Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Docker containers get incorrect HTTP_PROXY injected if the host has HTTP_PROXY=

See original GitHub issue

Describe the bug Running the runner in an environment such that the HTTP proxy runs on the same host as the runner, and the HTTP_PROXY et all environment variables are set to, results in docker containers being configured to have their HTTP_PROXY et all environment variables set to But docker containers by default run in the default bridge network, so connecting to from container basically always results in connection refused errors.

You could argue running the HTTP proxy on the same host is a dumb idea, because it means that docker containers cannot access the HTTP proxy, thereby breaking any docker actions that need to use the proxy. But in my environment of self-hosted runners we do the following: run a local HTTP forward proxy (that forwards to a parent proxy) as a docker container with a port published to the host, and set HTTP_PROXY et all environment variables of any other docker containers (on the default bridge network) by configuring $HOME/.docker/config.json (see docs). This way we can centrally configure credentials for the parent proxy and we can centrally configure a proxy bypass list. Without the central HTTP forward proxy we would require that any possible tools running in jobs:

  1. Support the authentication scheme to the parent proxy.
  2. Consistently interpret the proxy bypass list (but many tools don’t support CIDR notation, for example). Since our local HTTP forward proxy is written in Go we support CIDR notation in the proxy bypass list.

Running the proxy not on the same host, but on another compute unit, is not an ideal alternative because of 1 (authentication to the proxy).

It seems that docker container proxy configuration configured in $HOME/.docker/config.json (see docs) is always overridden.

Recently many workflows on self-hosted runners broke because they relied on pre behavior.

To Reproduce Steps to reproduce the behavior:

  1. Run a runner with a HTTP proxy set to
  2. Run a step with uses: docker:// syntax that needs to use the proxy.
  3. Observe that the container of the step gets a connection refused error.

Expected behavior If the runner’s HTTP_PROXY et all environment variables are set to localhost or (and/or if $HOME/.docker/config.json, docs is configured then docker containers do not explicitly get HTTP_PROXY set.

Runner Version and Platform

Version of your runner?

OS of the machine running the runner? OSX/Windows/Linux/… Linux

What’s not working?

A step with uses: docker:// such that the container does an HTTP request via the HTTP proxy configured in HTTP_PROXY gets a connection refused error because HTTP_PROXY is set to

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:3
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

richard-baincommented, Sep 14, 2021

We have a work around for this issue. The following assumes your runners are linux based and your proxy is running in a container:

  1. Find the gateway IP for your host: docker network inspect bridge -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}')
  2. Use that in your http/s proxy env variables on the host
  3. Ensure this gateway ip is in no_proxy

This works because from the host’s perspective this is a loopback to itself e.g. and a container on both the default bridge network and also any user-defined bridge network can also access this address.

Going one step further. if you are able to agree to a convention with your users you could do something like add no_proxy=local,... and remove the .local domain from the proxy. If everyone is prepared to do something like docker run ... --network-alias=<name>.local then this no proxy would also allow name based user-defined networking to work without having to modify the proxy every time.

FWIW I raised an upstream issue on libnetwork but don’t really expect action.

brekelj1commented, Sep 14, 2021

Closing due to the elegant workaround discovered by @richard-bain

Read more comments on GitHub >

github_iconTop Results From Across the Web

Docker connection refused between two containers
The thing is im trying to use n1 with nginx as a proxy redirect which is doing its job. Although when redirecting to...
Read more >
Docker Container Refuses to NOT use Proxy for ...
I've correctly configured (I think) Docker to get around the proxy for downloading images, but now my container is having trouble talking to ......
Read more >
Docker containers can't resolve DNS on Ubuntu 14.04 ...
First thing to check is run cat /etc/resolv.conf in the docker container. If it has an invalid DNS server, such as nameserver 127.0....
Read more >
Traefik Docker Documentation
Traefik retrieves the private IP and port of containers from the Docker API. ... is attacked, then the attacker might get access to...
Read more >
Docker Error Bind: Address Already in Use
In this tutorial, we'll learn to resolve the issues related to the port binding on a Docker container. This is one of the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found