Indefinite hanging of execution

It appears that the combined use of multithreading + requests + worker clients can cause an indefinite hanging of execution.

Setup

Running dask-scheduler with two dask-worker tcp://10.0.0.13:8786 --nthreads 1 workers. I can also reproduce with 2 threads so I don’t think this is super relevant, but you never know.

Code

Apologies for the rather complicated reproducible example.

import requests
import threading
import time

from distributed import Client, worker_client


class Heartbeat(threading.Timer):
    def run(self):
        self.finished.wait(self.interval)
        while not self.finished.is_set():
            self.function(*self.args, **self.kwargs)
            self.finished.wait(self.interval)
            

def heartbeat():   
    print('=' * 50)
    print('HEARTBEAT')
    requests.post("http://www.google.com")
    print('=' * 50)
    
    
def sleeper(secs):
    timer = Heartbeat(5.0, heartbeat)
    timer.daemon = True
    timer.start()
    
    for n in range(1, secs):
        requests.post("https://www.google.com")
    
    if not timer.is_alive():
        raise ValueError("heartbeat died!!")
    timer.cancel()
    timer.join()

    
def run():
    with worker_client() as c:
        futs = [c.submit(sleeper, i) for i in range(3, 10)]
        return c.gather(futs)
        
        
client = Client("tcp://10.0.0.13:8786")
result = client.gather(client.submit(run))
print('COMPLETE')

Symptoms

Whenever I run this code, it appears that:

• the execution never terminates
• at least one task usually continues heartbeating for a while (as evidenced by print statements)
• eventually, even this stops and I see:

==================================================
HEARTBEAT

with no ending print

• using something like time.sleep as the submitted function doesn’t reproduce the issue
• using separate_thread=False in the worker_client appears to fix this particular example, but “in the wild” I’m told this leads to a scheduler deadlock

This could could be a requests bug, but to be honest I’m not sure how to even begin digging deeper into this, or whether there’s a patch I could implement to avoid this hang.

[mrocklin]

I honestly have no idea. My guess is that this will come down to some hidden state within requests. But I would only put the odds of that at around 30-40%

Another approach would be to try to avoid threads altogether. I imagine that this is just a toy example for something that is more complex, but if you really wanted to post something somewhere at a regular interval I would probably use a tornado or asyncio HTTPClient rather than requests so that everything could live on the same thread. That tends to make everything safer.

You already know this, so I suspect that you genuinely do need a thread, but I thought I’d mention it anyway.

[cicdw]

OK so it appears that replacing my Heartbeat class with

import concurrent.futures
import time

class Heartbeat:
    def __init__(self, interval, function):
        self.interval = interval
        self.function = function

    def start(self):
        self.executor = concurrent.futures.ThreadPoolExecutor(max_workers=2)
        self._exit = False
        def looper():
            while not self._exit:
                self.function()
                time.sleep(self.interval)
        self.fut = self.executor.submit(looper)
 
   def exit(self):
        self._exit = True
        self.executor.shutdown()

Appears to prevents the hang, at the cost of losing the ability to “cancel” or kill the background thread while it’s sleeping.

What would make the ThreadPoolExecutor better than utilizing a Thread directly? Was my subclassed Timer poorly designed? Apologies for the questions, I’m just trying to figure out what to take away / learn from this.