multiprocessing failing when function contains a class on the same file
See original GitHub issueProblem Description
The following code executes properly from the command line but not within spyder. The program hangs and the command window show the traceback pasted below. Placing the class code (everything before if __name__ == "__main__")
in a separate file and importing the class solves the problem - but it is not clear why this code does not work as it does when executing from the command line.
What steps reproduce the problem?
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from multiprocessing import Pool
class myClass():
def __init__(self,q_seed=0,*args,**kwargs,) :
self.q_seed = q_seed
print(self.q_seed)
def simulateLocPool(kwargs) :
m = myClass(**kwargs)
return m
#%%
if __name__ == "__main__":
nboots = 10
nprocs = 10
# construct list to pass to processes
iterlist = []
kwargs = {'q_seed' : 0}
# change seeds
for bootn in range(nboots):
newkw = kwargs.copy()
newkw['q_seed'] = kwargs['q_seed'] + bootn
iterlist.append((newkw))
#spawn nboots processes, nprocs at once
pool = Pool(processes=nprocs)
allmodels = pool.map(simulateLocPool,(iterlist))
pool.close()
pool.join()
Output
The code hangs with the following traceback
Exception in thread Thread-23:
Traceback (most recent call last):
File "/Users/moroa/opt/miniconda3/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/Users/moroa/opt/miniconda3/lib/python3.9/threading.py", line 892, in run
self._target(*self._args, **self._kwargs)
File "/Users/moroa/opt/miniconda3/lib/python3.9/multiprocessing/pool.py", line 576, in _handle_results
task = get()
File "/Users/moroa/opt/miniconda3/lib/python3.9/multiprocessing/connection.py", line 256, in recv
return _ForkingPickler.loads(buf.getbuffer())
AttributeError: Can't get attribute 'myClass' on <module 'spyder_kernels.console.__main__' from '/Users/moroa/opt/miniconda3/lib/python3.9/site-packages/spyder_kernels/console/__main__.py'>
Expected output
A printout of digits from 0 to 9
Versions
- Spyder 5.1.5|Python 3.9.5 64-bit | Qt 5.12.10 | PyQt5 5.12.3 | Darwin 20.6.0
- Operating System name/version: MacOS 11.5.2
Issue Analytics
- State:
- Created 2 years ago
- Comments:9 (7 by maintainers)
Top Results From Across the Web
Python multiprocessing error with class methods
The pickle module can't serialize lambda functions because they all have the same name ( <lambda> ). Just use a conventional function and...
Read more >3 Multiprocessing Common Errors - Super Fast Python
Error 1: RuntimeError Starting New Processes; Error 2: print() Does Not Work In Child Processes; Error 3: Adding Attributes to Classes that ...
Read more >multiprocessing — Process-based parallelism — Python 3.11 ...
Context objects have the same API as the multiprocessing module, and allow one to use multiple ... The Pool class represents a pool...
Read more >Exception Handling in Methods of the Multiprocessing Pool ...
File "/usr/lib/python3.9/multiprocessing/pool.py", line 125, in worker ... to our function the printing of messages that this process has started and I will ...
Read more >Multiprocessing best practices — PyTorch 1.13 documentation
It supports the exact same operations, but extends it, so that all tensors sent through a multiprocessing.Queue , will have their data moved...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
I don’t know why this is not working, but
multiprocessing
could have problems pickling code in that situation.Thanks @impact27 for your help with this!
This will be fixed in our next version (5.2.2), to be released next week.