no live upstreams while connecting to upstream | error 500 & 503
See original GitHub issueI am tired of solving a problem, for 2 weeks, I spend a lot of time solving a problem of a proxy that was in operation for more than 2 years.
I have changed the docker-compose files to the latest version that appears in the documentation, but this has not generated a positive change, so I request your valuable support.
the problem is that when trying to access some site, nginx returns a 500 or a 503 error.
this is the nginx container configuration:
version: '2'
services:
nginx-proxy:
image: nginxproxy/nginx-proxy
container_name: nginx-proxy
restart: always
ports:
- "80:80"
- "443:443"
volumes:
- conf:/etc/nginx/conf.d
- vhost:/etc/nginx/vhost.d
- html:/usr/share/nginx/html
- dhparam:/etc/nginx/dhparam
- certs:/etc/nginx/certs:ro
- /var/run/docker.sock:/tmp/docker.sock:ro
network_mode: bridge
docker-gen:
image: nginxproxy/docker-gen
container_name: nginx-proxy-gen
command: -notify-sighup nginx-proxy -watch /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
restart: always
volumes_from:
- nginx-proxy
volumes:
- ./nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
- /var/run/docker.sock:/tmp/docker.sock:ro
network_mode: bridge
acme-companion:
image: nginxproxy/acme-companion
container_name: nginx-proxy-acme
restart: always
volumes:
- certs:/etc/nginx/certs:rw
- acme:/etc/acme.sh
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes_from:
- nginx-proxy
depends_on:
- "nginx-proxy"
network_mode: bridge
environment:
DEFAULT_EMAIL: server@example.com
NGINX_DOCKER_GEN_CONTAINER: nginx-proxy-gen
whoami:
image: jwilder/whoami
restart: always
expose:
- "8000"
environment:
- VIRTUAL_HOST=whoami.local
- VIRTUAL_PORT=8000
volumes:
conf:
vhost:
html:
dhparam:
certs:
acme:
networks:
default:
external:
name: nginx-proxy
this is the client container
version: '2' # version of docker-compose to use
services: # configuring each container
TS-DB1: # name of our mysql container
image: mariadb:latest # which image to pull, in this case specifying v. 5.7
volumes: # data to map to the container
- ./database/:/var/lib/mysql # where to find our data -- we'll talk more about this
restart: always # always restart the container after reboot
environment: # environment variables -- mysql options in this case
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: dbname
MYSQL_USER: dbuser
MYSQL_PASSWORD: password
TS-WP1: # name of our wordpress container
depends_on: # container dependencies that need to be running first
- TS-DB1
image: wordpress:latest # image used by our container
restart: always
environment:
VIRTUAL_HOST: example.com, www.example.com
VIRTUAL_PORT: 8003
LETSENCRYPT_HOST: www.example.com,example.com,cloud.example.com,tienda.example.com
LETSENCRYPT_EMAIL: server@example.com
WORDPRESS_DB_HOST: TS-DB1:3306 # default mysql port
WORDPRESS_DB_NAME: dbname # default mysql port
WORDPRESS_DB_USER: dbuser# default mysql port
WORDPRESS_DB_PASSWORD: password # matches the password set in the db containe
volumes: # this is where we tell Docker what to pay attention to
- ./html:/var/www/html # mapping our custom theme to the container
- ./php.ini:/usr/local/etc/php/conf.d/uploads.ini
networks:
default:
external:
name: nginx-proxy
When I create nginx with docker-compose it dies after a few seconds with this message:
nginx-proxy | WARNING: /etc/nginx/dhparam/dhparam.pem was not found. A pre-generated dhparam.pem will be used for now whil
e a new one
nginx-proxy | is being generated in the background. Once the new dhparam.pem is in place, nginx will be reloaded.
nginx-proxy | forego | starting dockergen.1 on port 5000
nginx-proxy | forego | starting nginx.1 on port 5100
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: using the "epoll" event method
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: nginx/1.21.0
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: built by gcc 8.3.0 (Debian 8.3.0-6)
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: OS: Linux 4.15.0-144-generic
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: getrlimit(RLIMIT_NOFILE): 1048576:10485
76
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: start worker processes
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: start worker process 40
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: start worker process 41
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: start worker process 42
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: start worker process 43
nginx-proxy | dockergen.1 | 2021/06/20 19:33:23 Generated '/etc/nginx/conf.d/default.conf' from 4 conta
iners
nginx-proxy | dockergen.1 | 2021/06/20 19:33:23 Running 'nginx -s reload'
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: signal 1 (SIGHUP) received from 45, rec
onfiguring
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: reconfiguring
nginx-proxy | dockergen.1 | 2021/06/20 19:33:23 Watching docker events
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: using the "epoll" event method
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: start worker processes
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: start worker process 48
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: start worker process 49
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: start worker process 50
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: start worker process 51
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 41#41: gracefully shutting down
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 41#41: exiting
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 42#42: gracefully shutting down
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 42#42: exiting
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 40#40: gracefully shutting down
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 40#40: exiting
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 42#42: exit
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 40#40: exit
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 43#43: gracefully shutting down
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 43#43: exiting
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 43#43: exit
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 41#41: exit
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: signal 17 (SIGCHLD) received from 40
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: worker process 40 exited with code 0
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: worker process 42 exited with code 0
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: signal 29 (SIGIO) received
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: signal 17 (SIGCHLD) received from 43
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: worker process 43 exited with code 0
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: signal 29 (SIGIO) received
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: signal 17 (SIGCHLD) received from 41
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: worker process 41 exited with code 0
nginx-proxy | nginx.1 | 2021/06/20 19:33:23 [notice] 34#34: signal 29 (SIGIO) received
nginx-proxy | dockergen.1 | 2021/06/20 19:33:24 Contents of /etc/nginx/conf.d/default.conf did not chan
ge. Skipping notification 'nginx -s reload'
nginx-proxy | dockergen.1 | 2021/06/20 19:33:24 Received event start for container c61e5ec61dd2
nginx-proxy | dockergen.1 | 2021/06/20 19:33:24 Received event start for container 9e41f8f62cfd
nginx-proxy | forego | sending SIGTERM to dockergen.1
nginx-proxy | forego | sending SIGTERM to nginx.1
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 48#48: signal 15 (SIGTERM) received from 1, exiti
ng
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 50#50: signal 15 (SIGTERM) received from 1, ex
iting
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 48#48: exiting
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 50#50: exiting
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 48#48: exit
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 50#50: exit
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 34#34: signal 15 (SIGTERM) received from 1, ex
iting
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 49#49: signal 15 (SIGTERM) received from 1, ex
iting
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 49#49: exiting
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 49#49: exit
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 51#51: signal 15 (SIGTERM) received from 1, ex
iting
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 51#51: exiting
nginx-proxy | nginx.1 | 2021/06/20 19:33:24 [notice] 51#51: exit
nginx-proxy | dockergen.1 | 2021/06/20 19:33:24 Received signal: terminated
nginx-proxy | dockergen.1 | 2021/06/20 19:33:24 Received signal: terminated
When trying to access a site from port 80 or 443, the following error appears:
by 80 port:
nginx-proxy | nginx.1 | example.com 192.168.1.231 - - [20/Jun/2021:20:38:51 +0000] "GET / HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0" "example.com-upstream"
nginx-proxy | nginx.1 | 2021/06/20 20:38:51 [error] 45#45: *42 no live upstreams while connecting to upstream, client: 192.168.1.231, server: example.com, request: "GET / HTTP/1.1", upstream: "http://example.com-upstream/", host: "example.com"
by 443 port:
nginx-proxy | nginx.1 | example.com 192.168.1.231 - - [20/Jun/2021:20:39:31 +0000] "GET / HTTP/2.0" 500 177 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0" "-"
nginx-proxy | nginx.1 | example.com 192.168.1.231 - - [20/Jun/2021:20:39:31 +0000] "GET /favicon.ico HTTP/2.0" 500 177 "https://example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0" "-"
when I try to get a new ssl certificate with docker exec nginx-proxy-acme /app/force_renew
I get this:
nginx-proxy | nginx.1 | cloud.example.com 34.221.255.206 - - [20/Jun/2021:20:44:28 +0000] "GET /.well-known/acme-challenge/EwjFpSqhqkuOcGVcPDwpE1HoPYOr8CFQlmaIUYWVj7g HTTP/1.1" 503 190 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
nginx-proxy | nginx.1 | cloud.example.com 3.142.122.14 - - [20/Jun/2021:20:44:28 +0000] "GET /.well-known/acme-challenge/EwjFpSqhqkuOcGVcPDwpE1HoPYOr8CFQlmaIUYWVj7g HTTP/1.1" 503 190 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
nginx-proxy | nginx.1 | cloud.example.com 66.133.109.36 - - [20/Jun/2021:20:44:28 +0000] "GET /.well-known/acme-challenge/EwjFpSqhqkuOcGVcPDwpE1HoPYOr8CFQlmaIUYWVj7g HTTP/1.1" 503 190 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
nginx-proxy | nginx.1 | cloud.example.com 18.184.29.122 - - [20/Jun/2021:20:44:28 +0000] "GET /.well-known/acme-challenge/EwjFpSqhqkuOcGVcPDwpE1HoPYOr8CFQlmaIUYWVj7g HTTP/1.1" 503 190 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
nginx-proxy | nginx.1 | tienda.example.com 66.133.109.36 - - [20/Jun/2021:20:44:33 +0000] "GET /.well-known/acme-challenge/sv7DLBk-Rp79GWz0oXno8JfRdtDdAQevJ9OumrChdCc HTTP/1.1" 503 190 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
nginx-proxy | nginx.1 | tienda.example.com 52.39.4.59 - - [20/Jun/2021:20:44:33 +0000] "GET /.well-known/acme-challenge/sv7DLBk-Rp79GWz0oXno8JfRdtDdAQevJ9OumrChdCc HTTP/1.1" 503 190 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
and the foce renew result this:
docker exec nginx-proxy-acme /app/force_renew
Creating/renewal www.example.com certificates... (www.example.com example.com cloud.example.com tienda.example.com)
[Sun Jun 20 20:44:13 UTC 2021] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sun Jun 20 20:44:13 UTC 2021] Creating domain key
[Sun Jun 20 20:44:21 UTC 2021] The domain key is here: /etc/acme.sh/server@example.com/www.example.com/www.example.com.key
[Sun Jun 20 20:44:22 UTC 2021] Multi domain='DNS:www.example.com,DNS:example.com,DNS:cloud.example.com,DNS:tienda.example.com'
[Sun Jun 20 20:44:22 UTC 2021] Getting domain auth token for each domain
^B[Sun Jun 20 20:44:26 UTC 2021] Getting webroot for domain='www.example.com'
[Sun Jun 20 20:44:26 UTC 2021] Getting webroot for domain='example.com'
[Sun Jun 20 20:44:26 UTC 2021] Getting webroot for domain='cloud.example.com'
[Sun Jun 20 20:44:26 UTC 2021] Getting webroot for domain='tienda.example.com'
[Sun Jun 20 20:44:27 UTC 2021] www.example.com is already verified, skip http-01.
[Sun Jun 20 20:44:27 UTC 2021] example.com is already verified, skip http-01.
[Sun Jun 20 20:44:27 UTC 2021] Verifying: cloud.example.com
[Sun Jun 20 20:44:30 UTC 2021] cloud.example.com:Verify error:Invalid response from http://cloud.example.com/.well-known/acme-challenge/EwjFpSqhqkuOcGVcPDwpE1HoPYOr8CFQlmaIUYWVj7g [IP-PUBLIC]:
[Sun Jun 20 20:44:30 UTC 2021] Please check log file for more details: /dev/null
My environtment configuration:
$ docker-compose -version
docker-compose version 1.29.2, build 5becea4c
$ docker -v
Docker version 20.10.7, build f0df350
$ docker -v
Docker version 20.10.7, build f0df350
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
b4295e60714a bridge bridge local
4728bf16f693 host host local
fdc61b1b1480 nginx-proxy bridge local
2e0bc41b39f7 none null local
$ uname -a
Linux serverhttp 4.15.0-144-generic #148-Ubuntu SMP Sat May 8 02:33:43 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ free
total used free shared buff/cache available
Mem: 3930672 392052 2879056 4356 659564 3320632
Swap: 2097148 0 2097148
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 1932496 0 1932496 0% /dev
tmpfs 393068 1492 391576 1% /run
/dev/sda2 76395292 20924768 51546764 29% /
tmpfs 1965336 0 1965336 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 1965336 0 1965336 0% /sys/fs/cgroup
/dev/loop0 89088 89088 0 100% /snap/core/4917
/dev/loop1 89984 89984 0 100% /snap/core/5742
/dev/loop2 90368 90368 0 100% /snap/core/5897
overlay 76395292 20924768 51546764 29% /var/lib/docker/overlay2/a9d62074a9ac482884984df110e1a3eea05a34b592f8f6456deb57b85e526391/merged
overlay 76395292 20924768 51546764 29% /var/lib/docker/overlay2/7abe60ee34ecdae6c4a56de63aedd626bcb81ddffec0f1a17d942a39782dca56/merged
tmpfs 393064 0 393064 0% /run/user/1000
overlay 76395292 20924768 51546764 29% /var/lib/docker/overlay2/d27753aa5a6b2234b83a327dd94e02dc26b6813cd5b78b9fc192a44292b327ff/merged
overlay 76395292 20924768 51546764 29% /var/lib/docker/overlay2/4a4fb8b03de81bc4666d93911f0ba31db2b35dcf95c9af304ff30c56c6bbf532/merged
overlay 76395292 20924768 51546764 29% /var/lib/docker/overlay2/47327e53cbd26df9b76c45bafeaef4f95c06726ba0ca1fa51a20e5ae0c6c33db/merged
overlay 76395292 20924768 51546764 29% /var/lib/docker/overlay2/76f276b83f73c21c663d3af5f60c0a00b32c613e1d35db0a08a46e675caf065d/merged
the result of this:
docker inspect yournginxproxycontainer
docker exec yournginxproxycontainer nginx -T
docker exec yournginxproxycontainer cat /proc/1/cpuset
docker exec yournginxproxycontainer cat /proc/self/cgroup
docker exec yournginxproxycontainer cat /proc/self/mountinfo
I don’t know what happened, since these services have worked for more than 2 years and now they are no longer working.
I hope you can guide me.
best regards.
Issue Analytics
- State:
- Created 2 years ago
- Comments:11 (6 by maintainers)
Hello @buchdag
I have solved this problem, but it is not the best practice, because I have downgraded the version of nginx-proxy to version 0.8.0 and again the backend containers are working correctly
I would like to contribute to this project and find out what is happening with version 0.9.0 and 0.9.1, friend @buchdag please tell me what I need to send to continue testing the latest versions of the project.
About your doubts to my variables, I answer each one.
I had added this variable to validate that the nginx containers were connected to the same network, but it never helped or made a correction to the problem, so it has been removed
after several errors this is my best configuration.
to each of the WP sites with different domain and subdomain, I have seen that for the VIRTUAL_HOST variable: I must add only the addresses that correspond to that container, for the LETSENCRYPT_HOST variable I must put all the domains and sub-domains that the bot Acme must validate, if I do not add all and in the same order, the bot that generates the new certificate will send an error, likewise all sites must be online so that a valid certificate can be generated
**This variable is the one that performs the communication of each WP site, after my tests I have seen that each of my WP site containers have a different port.
So far this configuration has worked for me and the sites work correctly**
Friend, I will leave this issue still open to be able to perform the tests with version 0.9.0 and 0.9.1
I don’t know much about how this containers works, but I will gladly provide whatever the project needs.
Regards
The issue was most probably the bogus
VIRTUAL_PORT
environment variables.Since this commit, the
VIRTUAL_PORT
is enforced even when it’s incorrect, which can result in non working configuration.That in turn prevented certificate issuance with
nginxproxy/acme-companion
.