Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
no servers inside upstream in /etc/nginx/conf.d/default.conf
See original GitHub issueDescription
On my x86 fedora 33 server environment I face the following issue with latest nginx-proxy image. The automatic default.conf generation seems to fail and results in an empty upstream configuration.
Steps to Reproduce
I use this minimal example:
version: '2'
services:
nginx-proxy:
image: jwilder/nginx-proxy
ports:
- "80:80"
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
whoami:
image: jwilder/whoami
environment:
- VIRTUAL_HOST=whoami.local
Startup log:
admin@tpdachsserver:~/repos/reverse-proxy-test ▶ docker-compose up
Creating network "reverse-proxy-test_default" with the default driver
Creating reverse-proxy-test_nginx-proxy_1 ... done
Creating reverse-proxy-test_whoami_1 ... done
Attaching to reverse-proxy-test_whoami_1, reverse-proxy-test_nginx-proxy_1
whoami_1 | Listening on :8000
nginx-proxy_1 | WARNING: /etc/nginx/dhparam/dhparam.pem was not found. A pre-generated dhparam.pem will be used for now while a new one
nginx-proxy_1 | is being generated in the background. Once the new dhparam.pem is in place, nginx will be reloaded.
nginx-proxy_1 | forego | starting dockergen.1 on port 5000
nginx-proxy_1 | forego | starting nginx.1 on port 5100
nginx-proxy_1 | dockergen.1 | 2021/01/02 03:07:54 Generated '/etc/nginx/conf.d/default.conf' from 2 containers
nginx-proxy_1 | dockergen.1 | 2021/01/02 03:07:54 Running 'nginx -s reload'
nginx-proxy_1 | dockergen.1 | 2021/01/02 03:07:54 Error running notify command: nginx -s reload, exit status 1
nginx-proxy_1 | dockergen.1 | 2021/01/02 03:07:54 Watching docker events
nginx-proxy_1 | dockergen.1 | 2021/01/02 03:07:54 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
nginx-proxy_1 | 2021/01/02 03:07:55 [emerg] 47#47: no servers are inside upstream in /etc/nginx/conf.d/default.conf:58
nginx-proxy_1 | nginx: [emerg] no servers are inside upstream in /etc/nginx/conf.d/default.conf:58
nginx-proxy_1 | Generating DSA parameters, 4096 bit long prime
nginx-proxy_1 | dhparam generation complete, reloading nginx
nginx-proxy_1 | nginx.1 | 172.28.0.1 - - [02/Jan/2021:03:11:18 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.71.1" "-"
As a consequence curl request returns the following:
Curl request
admin@tpdachsserver:~ ▶ curl -H "Host: whoami.local" localhost
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Please expand to see the resulting configuration:
default.conf
admin@tpdachsserver:~/repos/reverse-proxy-test $ docker exec -it reverse-proxy-test_nginx-proxy_1 cat /etc/nginx/conf.d/default.conf
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
default $http_x_forwarded_proto;
'' $scheme;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
default $http_x_forwarded_port;
'' $server_port;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
default upgrade;
'' close;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header
map $scheme $proxy_x_forwarded_ssl {
default off;
https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
access_log off;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
ssl_prefer_server_ciphers off;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
server_name _; # This is just an invalid value which will never trigger on a real hostname.
listen 80;
access_log /var/log/nginx/access.log vhost;
return 503;
}
# whoami.local
upstream whoami.local {
}
server {
server_name whoami.local;
listen 80 ;
access_log /var/log/nginx/access.log vhost;
location / {
proxy_pass http://whoami.local;
}
}
Pre-Analysis
I found one common issue which often results in this behaviour. It needs to be ensured the started containers are really in the same network and can reach each other. I tried many variants with external network and manually adding the containers to that network etc. All without success.
The second hint I found is that in some cases cgroup might be a reason .Docker.CurrentContainerID, used in nginx.tmpl, can’t be resolved. Since Fedora33 uses cgroup2 there might be a link. However, I lack the debug knowledge on how to check whether .Docker.CurrentContainerID is empty or not. I tried this very same example on an Debian Stretch server with 4.9 Kernel. Here the default.conf gets configured correctly. Thus, I assume some relation to the host environment.
Please let me know the supported Information is not sufficient. I’m grateful for any kind of support. TIA
Thanks, Steffen
Update #1
After some investigation I found that CurrentContainerID is empty and docker-gen can not get the information to fill the template. I prepared a RFC PR and a preliminary fix here: https://github.com/jwilder/docker-gen/pull/335
Issue Analytics
- State:
- Created 3 years ago
- Reactions:17
- Comments:31 (6 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
I’m still having the issue 😕
Some relevant info:
docker inspect nginx-proxy
Docker version:
Docker compose example:
I have the same issue and tried the fix by building myself the following image for the proxy (I changed the DOCKER_GEN_VERSION=0.7.5 to match the last tagged version which looks like embed the patch supposed to fix ):
But got the same upstream error Here is my docker info
EDIT: It looks like the upstream in default.conf is indeed empty where I run it locally it’s filled, so maybe linked to Arch/Docker swarm