`preprocess` fails for `n_jobs`>1
See original GitHub issueRunning the latest master I run into issues with the parallelised preprocessing (from #277):
from braindecode.preprocessing import Preprocessor, preprocess, filterbank
from braindecode.datasets.moabb import MOABBDataset
ds = MOABBDataset(dataset_name='Schirrmeister2017',
subject_ids=[1], )
splits = ds.split([[i] for i in range(len(ds.datasets))])
fbank = [
Preprocessor(
apply_on_array=False,
fn=filterbank,
frequency_bands=[(0, 10)],
verbose=False,
n_jobs=1
),
]
for split_i, split_ds in splits.items():
preprocess(concat_ds=split_ds, preprocessors=fbank, n_jobs=2)
gives the following output:
Creating RawArray with float64 data, n_channels=128, n_times=1225545
Range : 0 ... 1225544 = 0.000 ... 2451.088 secs
Ready.
Creating RawArray with float64 data, n_channels=128, n_times=616535
Range : 0 ... 616534 = 0.000 ... 1233.068 secs
Ready.
320 events found
Event IDs: [1 2 3 4]
160 events found
Event IDs: [1 2 3 4]
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 431, in _process_worker
r = call_item()
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 285, in __call__
return self.fn(*self.args, **self.kwargs)
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 595, in __call__
return self.func(*args, **kwargs)
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/joblib/parallel.py", line 262, in __call__
return [func(*args, **kwargs)
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/joblib/parallel.py", line 262, in <listcomp>
return [func(*args, **kwargs)
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/braindecode/preprocessing/preprocess.py", line 223, in _preprocess
_preprocess_raw_or_epochs(ds.raw, preprocessors)
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/braindecode/preprocessing/preprocess.py", line 220, in _preprocess_raw_or_epochs
preproc.apply(raw_or_epochs)
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/braindecode/preprocessing/preprocess.py", line 62, in apply
self._try_apply(raw_or_epochs)
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/braindecode/preprocessing/preprocess.py", line 74, in _try_apply
self.fn(raw_or_epochs, **self.kwargs)
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/braindecode/preprocessing/preprocess.py", line 473, in filterbank
raw.add_channels(all_filtered)
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/mne/channels/channels.py", line 1051, in add_channels
self._data.base.resize(n_bytes)
TypeError: mmap can't resize a readonly or copy-on-write memory map.
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-3-afd28816663e>", line 19, in <module>
preprocess(concat_ds=split_ds, preprocessors=fbank, n_jobs=2)
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/braindecode/preprocessing/preprocess.py", line 158, in preprocess
list_of_ds = Parallel(n_jobs=n_jobs)(
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/joblib/parallel.py", line 1054, in __call__
self.retrieve()
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/joblib/parallel.py", line 933, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "/home/dan/Documents/AGBALL/RiemNetEnv/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 542, in wrap_future_result
return future.result(timeout=timeout)
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
TypeError: mmap can't resize a readonly or copy-on-write memory map.
Issue Analytics
- State:
- Created 2 years ago
- Comments:11
Top Results From Across the Web
_pickle.PicklingError: Can't pickle <function <lambda ... - GitHub
PicklingError: Can't pickle at 0x7f253aa720d0>: attribute lookup on __main__ failed when executing ... preprocessor=lambda x: [re.sub(re.compile(r'\s{2,}'), ...
Read more >CNprep: Pre-Process DNA Copy Number (CN) Data for ...
Title Pre-Process DNA Copy Number (CN) Data for Detection of CN Events. Version 2.2 ... myseed = 123, distrib = c("vanilla", "Rparallel"), njobs...
Read more >CONN : functional connectivity toolbox: help - NITRC
I have the same problem when running CONN locally even with few subjects (<10), ... I have specified RUNPARALLEL=false and NJOBS=1 for serial...
Read more >FSL Archives - JISCMail
Hello, I've been trying to compile FSL so that I can run bedpostx_gpu with a single GPU on my desktop with Ubuntu 14.04.3...
Read more >C preprocessing fails to stop immediately after an #error
I have a draft copy of the C ISO spec, and in §4/4 - it states. The implementation shall not successfully translate a...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I was just able to reproduce the issue on a server, I’ll look into it @dcwil
Hi @bruAristimunha
Looks like I can still reproduce the error in 0.6, using the same snippet