Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Graceful shutdown unresponsive to shutdownGraceMs setting

See original GitHub issue

Filing a Linkerd issue

Issue Type:

Bug report

What happened: I was testing linker-to-linker architecture. Sent a sequence of non-stop requests to an endpoint behind a Linkerd proxy and restarted the proxy without stoping the requests. I witnessed that no matter the load, Linkerd get killed in about 2 seconds. I’ve searched after a setting for this period of time before termination, maybe explicitly configuring a larger interval. I’ve configured my admin block in the configuration file with shutdownGraceMs: 10000 but still - Linkerd is terminated after 2 seconds.

What you expected to happen: A) A more cohesive graceful shutdown, closing sockets/connections, draining the queue of requests until none is there and then terminating. B) Linkerd task would be terminated after the period of time configured in the admin block.

How to reproduce it (as minimally and precisely as possible):

---
admin:
  ip: 0.0.0.0
  port: 4447
  shutdownGraceMs: 10000

Then see exact time of SIGTERM and exit in stderr:

I 0329 08:14:48.002 UTC THREAD29 TraceId:b229fc3755c0e29f: [/admin/shutdown] from <TASK_IP> quitting
I0329 08:14:49.415153 18340 health_checker.cpp:165] Health checking stopped
W0329 08:14:49.415153 18337 logging.cpp:91] RAW: Received signal SIGTERM from process 5704 of user 0; exiting

Anything else we need to know?: I think that maybe this issue is caused by the graceful shutdown issue.

Environment:

linkerd/namerd version, config files: the only relevant config is in the admin block
Platform, version, and config files (Kubernetes, DC/OS, etc): DC/OS
Cloud provider or hardware configuration: AWS