question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

multiprocessing failing when function contains a class on the same file

See original GitHub issue

Problem Description

The following code executes properly from the command line but not within spyder. The program hangs and the command window show the traceback pasted below. Placing the class code (everything before if __name__ == "__main__") in a separate file and importing the class solves the problem - but it is not clear why this code does not work as it does when executing from the command line.

What steps reproduce the problem?

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from multiprocessing import Pool
class myClass():
    
    def __init__(self,q_seed=0,*args,**kwargs,) :
        self.q_seed = q_seed
        print(self.q_seed)

def simulateLocPool(kwargs) : 

    m = myClass(**kwargs)  
    return m   

#%%
if __name__ == "__main__":
    
    nboots = 10
    nprocs = 10
    
    # construct list to pass to processes
    iterlist = []
    kwargs = {'q_seed' : 0}
    
    # change seeds
    for bootn in range(nboots):
        newkw = kwargs.copy()
        newkw['q_seed'] = kwargs['q_seed'] + bootn
        iterlist.append((newkw)) 
    
    #spawn nboots processes, nprocs at once    
    pool = Pool(processes=nprocs)
    allmodels = pool.map(simulateLocPool,(iterlist))
    pool.close()
    pool.join()

Output

The code hangs with the following traceback


Exception in thread Thread-23:
Traceback (most recent call last):
  File "/Users/moroa/opt/miniconda3/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/Users/moroa/opt/miniconda3/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/moroa/opt/miniconda3/lib/python3.9/multiprocessing/pool.py", line 576, in _handle_results
    task = get()
  File "/Users/moroa/opt/miniconda3/lib/python3.9/multiprocessing/connection.py", line 256, in recv
    return _ForkingPickler.loads(buf.getbuffer())
AttributeError: Can't get attribute 'myClass' on <module 'spyder_kernels.console.__main__' from '/Users/moroa/opt/miniconda3/lib/python3.9/site-packages/spyder_kernels/console/__main__.py'>

Expected output

A printout of digits from 0 to 9

Versions

  • Spyder 5.1.5|Python 3.9.5 64-bit | Qt 5.12.10 | PyQt5 5.12.3 | Darwin 20.6.0
  • Operating System name/version: MacOS 11.5.2

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:9 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
ccordoba12commented, Nov 8, 2021

I don’t know why this is not working, but

  • If you’re running your code in an unsaved file (as @athompson673 seems to be doing), it could be related to this because multiprocessing could have problems pickling code in that situation.
  • The work @impact27 is doing in PR #14025 could help at least to debug these problems.
0reactions
ccordoba12commented, Jan 10, 2022

Thanks @impact27 for your help with this!

This will be fixed in our next version (5.2.2), to be released next week.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Python multiprocessing error with class methods
The pickle module can't serialize lambda functions because they all have the same name ( <lambda> ). Just use a conventional function and...
Read more >
3 Multiprocessing Common Errors - Super Fast Python
Error 1: RuntimeError Starting New Processes; Error 2: print() Does Not Work In Child Processes; Error 3: Adding Attributes to Classes that ...
Read more >
multiprocessing — Process-based parallelism — Python 3.11 ...
Context objects have the same API as the multiprocessing module, and allow one to use multiple ... The Pool class represents a pool...
Read more >
Exception Handling in Methods of the Multiprocessing Pool ...
File "/usr/lib/python3.9/multiprocessing/pool.py", line 125, in worker ... to our function the printing of messages that this process has started and I will ...
Read more >
Multiprocessing best practices — PyTorch 1.13 documentation
It supports the exact same operations, but extends it, so that all tensors sent through a multiprocessing.Queue , will have their data moved...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found