question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Cannot employ a Lambda transform into SubjectsDatasets for Queue

See original GitHub issue

🐛Bug

Hi guys, thanks for the library is quite cool!

I want to employ torchio in a 2D architecture so I am employing the Queue dataset in order to extract patches that represent one slice. Everything works fine when I do not apply transforms or I employ your built-in transforms. However, since I am using a pre-pre-trained model with a different slice orientation I am trying to apply an easy Lambda which just employs torch.transpose. In this case, I get the error showed at traces paragraph.

Just in case let me add that the Lambda transform alone works OK

To reproduce

transpose_transform = tio.Lambda(lambda x: torch.transpose(x, 1,2))
transformadas = tio.Compose([transpose_transform]) #Without or  with built-in transforms works properly
dataset_transform = tio.SubjectsDataset(subjects, transform=transformadas) # I tried with just the transpose (instead a composition) transform but I got the same result

slices_queue = tio.Queue(
    subjects_dataset=dataset_transform,
    max_length=24,
    samples_per_volume=4, # For this test I employ a datsset just with 6 3D-images 
    sampler=tio.sampler.UniformSampler((500,400,1)),
    num_workers=6,
    shuffle_subjects=False,
    shuffle_patches=True)

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 234, in _feed
    obj = _ForkingPickler.dumps(obj)
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 234, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f3d89c6fb70>: attribute lookup <lambda> on __main__ failed
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f3d89c6fb70>: attribute lookup <lambda> on __main__ failed
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 234, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f3d89c6fb70>: attribute lookup <lambda> on __main__ failed
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 234, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f3d89c6fb70>: attribute lookup <lambda> on __main__ failed
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 234, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f3d89c6fb70>: attribute lookup <lambda> on __main__ failed
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 234, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f3d89c6fb70>: attribute lookup <lambda> on __main__ failed

Expected behavior

Work as with the built-in transforms

“Can’t pickle”

TorchIO version 0.18.0

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:10 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
PeterMcGorcommented, Dec 1, 2020

Yeah!

0reactions
fepegarcommented, Jan 13, 2021

Hi, @Geeks-Sid. Could you please open a new issue and share a minimal, reproducible example that I can run on my Linux computer? That will help me solve your issue.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshooting Lambda configurations - AWS Documentation
Memory configurations. You can configure a Lambda function to use between 128 MB and 10,240 MB. By default, any function created in the...
Read more >
Processing Large S3 Files With AWS Lambda - Medium
Lambda is a good option if you want a serverless architecture and have files that are large but still within reasonable limits. It...
Read more >
Python multiprocessing PicklingError: Can't pickle <type ...
The problem is that the pool methods all use a mp.SimpleQueue to pass tasks to the worker processes. Everything that goes through the...
Read more >
3 Pitfalls of AWS Step Functions and How You Can Avoid Them
When starting out with Step Functions, it is entirely possible to use Lambda functions, SQS queues, SNS topics, etc… without regard to which ......
Read more >
Data — MONAI 1.1.0 Documentation
Extend the IterableDataset with a buffer and randomly pop items. Parameters. data – input data source to load and transform to generate dataset...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found