reuse address on server

start_hpp_server should reuse the port if it is already being used, most likely due to an earlier instance going down and we are restarting

File "lib64/python3.4/site-packages/prometheus_client/exposition.py", line 103, in run
    httpd = HTTPServer((addr, port), MetricsHandler)
  File "/usr/lib64/python3.4/socketserver.py", line 430, in __init__
    self.server_bind()
  File "/usr/lib64/python3.4/http/server.py", line 136, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/usr/lib64/python3.4/socketserver.py", line 444, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use

[itldo]

I’ve been living with my supervisord workaround described in my last post. I had made my proposed change in my own build, which appeared also to work, however I never took the time to submit it, as my workaround was meeting my needs. I need to consider doing that. -tim

[tommyjcarpenter]

A simple fix for this issue might be to have httpd as a global that can be accessed - we are doing something similar. That way the caller can call httpd.shutdown() in a signal handler. Something like:

httpd: Optional[HTTPServer] = None

def handler(signum: int, _frame: Any) -> None:
    """gracefully shutdown servers on signal"""
    signame = signal.Signals(signum).name
    logger.warn(f"Signal handler called with signal {signame} ({signum})")
    if httpd:
        httpd.shutdown()
        logger.info("Shutdown metrics server")

def start_http_server():
    global httpd
    httpd = HTTPServer(("", 8080), SomeHandler)
    server = threading.Thread(name="metrics", target=httpd.serve_forever)
    server.daemon = True
    server.start()
    logger.debug("Servers started")

client program:

# Set the signal handler
def __main__():
    ...
    signal.signal(signal.SIGINT, handler)  # ctrl-c
    signal.signal(signal.SIGTERM, handler)  # what kubernetes sends for graceful shutdown