Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Make it easier to extend joblib.Parallel with a custom backend

See original GitHub issue

I know this title is very generic but let me explain:

I am a developer for SageMath and I learned about joblib recently during some talk at the OSI day in Paris-Saclay (Orsay). One problem with Python is that the multiprocessing module sucks badly. It seems that joblib.Parallel tries to make the front-end more user-friendly by adding a better API. However, all the problems which really come from the multiprocessing.Pool backend are not addressed.

In SageMath, we have implemented a module to run doctest in parallel. It has a custom implementation of something like multiprocessing.Pool features using multiprocessing.Process. This works very well but it’s really a custom-built solution just for doctesting.

We would like to use multiprocessing (the principle, not the Python module) also in other places in SageMath. There are two options: either we make our custom-built parallel doctester into a proper Python package or we look for existing solutions (we don’t want to reinvent the wheel). For the latter, I looked at joblib.Parallel but I really don’t like the multiprocessing.Pool backend.

Since joblib.Parallel is more about providing a nice front-end and the SageMath parallel doctester is more about providing a solid backend, it would be useful to combine efforts and make it possible to allow joblib.Parallel to use a backend written by SageMath.

Note that the SageMath code is not 100% portable (it assumes POSIX) and has dependencies other than Python (in particular Cython), so it probably will not be possible to add the SageMath backend to the joblib codebase.

Issue Analytics

State:
Created 8 years ago
Comments:13 (7 by maintainers)

Top GitHub Comments

1reaction

hristogcommented, Sep 10, 2021

I could be wrong (I can see that the most recent comments here are from after the opening and merge of the latter) but to me it seems this can be closed, as a result of https://github.com/joblib/joblib/pull/306 having been merged a while ago.

0reactions

tomMoralcommented, Sep 10, 2021

Indeed, there is now an API to subclass the parallel backend: https://joblib.readthedocs.io/en/latest/parallel.html#custom-backend-api-experimental that was merged in #320

Let’s close this issue.

Top Results From Across the Web

Embarrassingly parallel for loops - Joblib - Read the Docs

Joblib provides a simple helper class to write parallel for loops using multiprocessing. The core idea is to write the code to be...

joblib - Parallel Processing in Python - CoderzColumn

It's a guide to using Joblib as a parallel programming/computing backend. Tutorial covers the API of Joblib with simple examples. It starts with ......

Python: Parallel Processing in Joblib Makes the Code Run ...

See below example where I am using joblib with a simple function on a list of random integers. Notice that without the parallel...

Use Joblib to run your Python code in parallel - Measure Space

As the increase of PC computing power, we can simply increase our computing by running parallel code in our own PC. Joblib is...

joblib Documentation - Read the Docs

2) Embarrassingly parallel helper: to make it easy to write ... Warning: The custom backend API is experimental and subject to change ...