server ip is marked as down when using different internal networks

Got a docker-compose file which will have multiple internal docker networks. The proxy seems to find the correct ip and set it to the correct upstream but the server is marked as down, eventhough the commet says nginx was able to connect to the right network:

default.conf:

#[...]
# cloud.website.com
upstream cloud.website.com {
				# Cannot connect to network of this container
				server 127.0.0.1 down;
				## Can be connected with "test_net-dashboard" network
		# test_nextcloud_1
			server 172.31.0.4 down;
}
server {
	server_name cloud.website.com;
	listen 80 ;
	access_log /var/log/nginx/access.log vhost;
	location / {
		proxy_pass http://cloud.website.com;
	}
}

When I remove the down, safe it and then reload nginx with docker exec -it test_proxy_1 nginx -s reload it works like a charm. But if a container starts/stops the file will be rewritten (this is why I want to use this tool…)

I created a simple docker-compose file, to show the problem:

version: '3'

services:
  db:
    image: postgres:alpine
    restart: always
    volumes:
      - db:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=
      - POSTGRES_DB=nextcloud
      - POSTGRES_USER=usr
    networks:
      - net-dashboard
  nextcloud:  
    image: nextcloud:apache
    restart: always
    volumes:
      - nextcloud:/var/www/html
    environment:
      - VIRTUAL_HOST=cloud.website.com
      - POSTGRES_HOST=db
      - MM_USERNAME=usr
      - MM_PASSWORD=
      - MM_DBNAME=nextcloud
    depends_on:
      - db
    networks:
     - net-dashboard
  #NGINX Proxy
  proxy:
    image: jwilder/nginx-proxy:alpine
    restart: always
    ports:
      - 80:80
    volumes:
#      - certs:/etc/nginx/certs:ro
      - vhost.d:/etc/nginx/vhost.d
      - conf.d:/etc/nginx/conf.d
      - html:/usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro
    networks:
      - net-dashboard
      - default
 

volumes:
  db:
  nextcloud:
#  certs:
  vhost.d:
  conf.d:
  html:


networks:
  net-dashboard:
    internal: true

This seems to be a bug, so I posted it here. Thanks in advance, Spider

[webdevotion]

Thanks @meron1122 for the pointer.

I can confirm that removing all internal networks from my app’s docker-compose.yml fixed it for me. My default.conf had similar lines with server 127.0.0.1 down; under the upstream entry.

After removing all internal networks and declaring the default, external network everything started working as expected. Mixing in the webproxy network together with other internal networks did not work for me.

For future Googlers.

Put the snippet from @meron1122’s comment at the bottom of your docker compose file and remove all other entries of networks in each of the container’s configuration. Please understand that there might be security implications as pointed out by @gtaspider.

Important notice

Don’t forget to create the external network webproxy. That network will be available for all your dockerized environments. You can do so by running:

$ docker network create webproxy

This might help

You might want to stop all your services ( nginx-proxy and your application’s stack ) before restarting them one at a time. First the nginx-proxy stack, then your application(s).

I am only using the webproxy network

No other networks are used anywhere in my docker-compose.yml file. I will probably try to get things working again with multiple, ( internal, ) networks to split up frontend, backend, … and so on. But his works for now. As you can read in the comments above, it seems to be quite a challenge to find a permanent fix.

[meron1122]

If you don’t need separate subnet, but you have separate docker-compose, simple workaround is setting up default network to join nginx network.

Add this of the bottom of your app docker-compose file:

networks:
  default:
    external:
      name: webproxy