nginx fails to bind to ports

I’m having trouble getting nginx to bind to port 80 and 443 in my production docker deployment. I’ve set up the cookiecutter with docker, letsencrypt, compressor, whitenoise, mailgun, sentry and opbeat. All seems to work fine, the letsencrypt key + cert gets generated, django runs, copies statics and compresses js, and the database starts up fine. Until I get to nginx trying to bind to port 80 and 443. It seems to be not finding a process to kill in nginx’s start.sh kill line? Here’s the last part of my docker-compose up attempt:

certbot_1   | Cert not yet due for renewal
certbot_1   | Keeping the existing certificate
certbot_1   |
certbot_1   | -------------------------------------------------------------------------------
certbot_1   | Certificate not yet due for renewal; no action taken.
certbot_1   | -------------------------------------------------------------------------------
xx_certbot_1 exited with code 0
nginx_1     | replacing ___my.example.com___/xx.com
nginx_1     | replacing ___NAMESERVER___/127.0.0.11
nginx_1     | /start.sh: 59: /start.sh: ps: not found
nginx_1     | /start.sh: 59: kill: Usage: kill [-s sigspec | -signum | -sigspec] [pid | job]... or
nginx_1     | kill -l [exitstatus]
nginx_1     | 2017/05/04 17:45:34 [emerg] 30#30: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1     | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1     | 2017/05/04 17:45:34 [emerg] 30#30: bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx_1     | nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx_1     | 2017/05/04 17:45:34 [emerg] 30#30: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1     | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1     | 2017/05/04 17:45:34 [emerg] 30#30: bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx_1     | nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx_1     | 2017/05/04 17:45:34 [emerg] 30#30: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1     | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1     | 2017/05/04 17:45:34 [emerg] 30#30: bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx_1     | nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx_1     | 2017/05/04 17:45:34 [emerg] 30#30: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1     | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1     | 2017/05/04 17:45:34 [emerg] 30#30: bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx_1     | nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx_1     | 2017/05/04 17:45:34 [emerg] 30#30: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1     | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1     | 2017/05/04 17:45:34 [emerg] 30#30: bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx_1     | nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx_1     | 2017/05/04 17:45:34 [emerg] 30#30: still could not bind()
nginx_1     | nginx: [emerg] still could not bind()
xx_nginx_1 exited with code 1

My .env, I’m unsure about this - there’s not a lot of documentation on it:

DJANGO_ADMIN_URL=r’^admin/’
DJANGO_SETTINGS_MODULE=config.settings.production
DJANGO_SECRET_KEY=censored
DJANGO_ALLOWED_HOSTS=.xx.com

# AWS Settings
DJANGO_AWS_ACCESS_KEY_ID=
DJANGO_AWS_SECRET_ACCESS_KEY=
DJANGO_AWS_STORAGE_BUCKET_NAME=

# Used with email
DJANGO_MAILGUN_API_KEY=key-xx
DJANGO_SERVER_EMAIL="xx Team <[e-mail]>"
MAILGUN_SENDER_DOMAIN=mg.xx.com

# Security! Better to use DNS for this task, but you can use redirect
DJANGO_SECURE_SSL_REDIRECT=False

# django-allauth
DJANGO_ACCOUNT_ALLOW_REGISTRATION=True
# Sentry
DJANGO_SENTRY_DSN=https://[e-mail]/xx

DJANGO_OPBEAT_ORGANIZATION_ID=xxb5
DJANGO_OPBEAT_APP_ID=xx8f
DJANGO_OPBEAT_SECRET_TOKEN=xx88

COMPRESS_ENABLED=True

I followed all the steps in the docs quite closely and this is an almost clean install of Ubuntu 16.04.2 LTS running docker 17.03.1 from their site, and docker-compose 1.13.0 (docker-py 2.2.1, CPython 2.7.13).

[oesah]

So I have a quick fix for now:

Comment out or delete the Lets encrypt block in nginx.conf:

    #location /.well-known/acme-challenge {
    #  proxy_pass http://certbot:80;
    #  proxy_set_header Host            $host;
    #  proxy_set_header X-Forwarded-For $remote_addr;
    #  proxy_set_header X-Forwarded-Proto https;
    #}

This solves the “certbot not in upstream” error.

In addition, you need to change the start.sh file in compose/nginx and change

kill $(ps aux | grep 'nginx' | grep -v 'grep' | awk '{print $2}')

to

nginx -s stop

This works for me, but I guess I lose the auto renewal of the Let’s encrypt certificate? Also it seems a bit hacky, I would love for the pros to come up with a solid solution. Also, it would be interesting to know why this does not work all of the sudden. It worked for almost a year for me just now it got messed up…

[jonesnc]

i’m still experiencing this. I don’t think the nginx process is getting properly terminated, since nginx is the main process in the container. If I try to manually perform the kill $(ps aux | grep 'nginx' | grep -v 'grep' | awk '{print $2}'), it kills the main nginx process, and therefore the nginx container itself dies.

I’m not sure how the proposed solution is supposed to be an actual solution to this.