question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Docker service on a node can reload config only twice

See original GitHub issue

When try to reload config of docker (without restart) on a Codenvy node more then two times changes are ignored.

Reproduction Steps:

  1. Run docker info and remember existing labels (if no labels configured section is not appeared)
  2. Edit / create file /etc/docker/daemon.json and add a new label. Example of content of created file: {"labels":["com.codenvy.some-label=somevalue"]}
  3. Reload docker config. Do NOT restart docker, use sighup signal: kill -SIGHUP $(<"/var/run/docker.pid") Use docker info to ensure, that the label is present.
  4. Change label value and reload config again. Check again, that changes are applied.
  5. Change label value again and again reload config (3rd time).

Expected behavior: Config should be reloaded again and next times.

Observed behavior: Nothing changes. Config is left as after second reloading. To apply new changes docker restart is needed.

Codenvy version: 5.0.0-M8-SNAPSHOT OS and version: CentOS Linux 7 (Core) 3.10.0-327.10.1.el7.x86_64 Docker version: Version: 1.11.1 API version: 1.23 Swarm: Server Version: swarm/1.2.3

Additional information:

Problem only started happening recently, didn’t happen in an older version of Codenvy: Yes Problem can be reliably reproduced: Yes

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Comments:8 (8 by maintainers)

github_iconTop GitHub Comments

2reactions
mmorhuncommented, Feb 17, 2017

Issue in docker is resolved, fix is merged for 1.14.0 milestone.

0reactions
mmorhuncommented, Feb 7, 2017

This problem occurs when docker on a node is started with cluster-store and cluster-advertise options. Otherwise everything works fine. In case, when only cluster-store is set, docker hungs up on 3rd config reload. Logs: after 1st SIGHUP:

ime="2017-02-07T13:19:18.661579317Z" level=info msg="Got signal to reload configuration, reloading from: /etc/docker/daemon.json" 
time="2017-02-07T13:19:18.661771835Z" level=debug msg="Initializing discovery service" name=zk uri="<host-ip>:2181" 
time="2017-02-07T13:19:18.661784409Z" level=info msg="Initializing discovery without TLS" 
time="2017-02-07T13:19:18.661811447Z" level=debug msg="Option Experimental: false" 
time="2017-02-07T13:19:18.661817781Z" level=debug msg="Option DefaultDriver: bridge" 
time="2017-02-07T13:19:18.661822227Z" level=debug msg="Option DefaultNetwork: bridge" 
time="2017-02-07T13:19:18.661826658Z" level=debug msg="Option OptionKVProvider: zk" 
time="2017-02-07T13:19:18.661831655Z" level=debug msg="Option OptionKVProviderURL: <host-ip>:2181" 
time="2017-02-07T13:19:18.661840320Z" level=debug msg="Reset Max Concurrent Downloads: 4" 
time="2017-02-07T13:19:18.661845701Z" level=debug msg="Reset Max Concurrent Uploads: 5" 

2nd:

time="2017-02-07T13:20:18.620647092Z" level=info msg="Got signal to reload configuration, reloading from: /etc/docker/daemon.json" 
time="2017-02-07T13:20:18.620767361Z" level=error msg="Error reconfiguring the daemon: discovery service not supported"

3rd:

time="2017-02-07T13:20:35.780443737Z" level=info msg="Got signal to reload configuration, reloading from: /etc/docker/daemon.json"
Read more comments on GitHub >

github_iconTop Results From Across the Web

Keep containers alive during daemon downtime
There are two ways to enable the live restore setting to keep containers alive when the daemon becomes unavailable. Only do one of...
Read more >
Deploy services to a swarm - Docker Documentation
First, create overlay network on a manager node using the docker network create command with the --driver overlay flag. After you create an...
Read more >
docker service create - Docker Documentation
Create a service with a config. The config will be mounted into redis-config , be owned by the user who runs the command...
Read more >
Docker daemon configuration overview
To configure the Docker daemon using a JSON file, create a file at /etc/docker/daemon. json on Linux systems, or C:\ProgramData\docker\config\daemon. json on ...
Read more >
docker service update - Docker Documentation
For example, only changing the --update-parallelism setting will not ... be used to perform a rolling restart without any changes to the service...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found