[Question] Consequence of broken semaphore on OSX?

Hello, this is not an issue, but I’m simply trying to understand the consequences of this warning that appears on OSX (due to the missing sem_getvalue function):

loky/backend/semlock.py:217: RuntimeWarning: semaphore are broken on OSX, release might increase its maximal value

import time
from loky import ProcessPoolExecutor

def f(x):
    time.sleep(6)
    return x*x


if __name__ == '__main__':
    with ProcessPoolExecutor(max_workers=3) as executor:
        print("Started")
        results = executor.map(f, [1, 2, 3])
        print(list(results))
        print("Ended")

What is the worse case outcome of this warning?

I also noticed that I could change process_executor.py to use a queue_size of 1 to avoid the warning:

        # Finally setup the queues for interprocess communication
        self._setup_queues(job_reducers, result_reducers, 1) #force a queue_size of 1

But I’m not sure what that would mean in terms of performance.

Love the work by the way, keep it up! ❤️

[tomMoral]

The consequence of the broken semaphore on OSX is that we are not able to guarantee that the size of the ProcessPoolExecutor._call_queue will stay the same, except when queue_size=1.

Indeed, the semaphore is an atomic counter, that you can increase or decrease atomically and which blocks in 0. If a thread tries to decrease it when it is 0, it waits until another thread increases it. The semaphore have an upper bound which is MAX_SEM_VALUE, depending on your OS.

When we define a semaphore in loky, we give a maximal value max_value, which can be lower than the global MAX_SEM_VALUE. We use sem_get_value to make sure we do not exceed this value. As this function is not implemented on OSX, we have no guarantees when releasing that we won’t increase the value of the semaphore over max_value. This is not a problem if max_value = MAX_SEM_VALUE, as the OS semaphores handle the case. But this means that there is no BoundedSemaphore in OSX.

If you are using the loky.ProcessPoolExecutor, this is not a big concern. The semaphore with max_value > 1 is the one controlling the size of the call_queue. On OSX, all the jobs will be en-queued at once, as the size of the Queue is not enforced. The size of the Queue is limited in ProcessPoolExecutor in order to allow you to cancel jobs. On OSX, you won’t be able to cancel jobs because as soon as you submit them, they will be put in queue and thus considered running. The concurrent.futures API does not permit to kill running Futures. If you have no need to cancel jobs, you can ignore this Warning.

Note that setting queue_size=1 is a bad idea for performances, as you will get a call_queue of size 1. This means that your executor will not have any buffer and your workers will be underfed.

[tomMoral]

This issue is inherent to OSX and does not depend on python version. OS X does not implement the function sem_getvalue which is optional in the POSIX standard. The warning only appears in python2 because we use a different implementation of the semaphores.

The behavior will not change unless OS X implement the primitive, or we try to come up with a platform specific fix, maybe using SysV semaphores and semctl.