question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Memory leak on GaussianBlur

See original GitHub issue

🐛 Describe the bug

Hello. When using num_workers > 0 for dataloader and GaussianBlur BEFORE the resize function in transforms (images in dataset are of different size) a memory leak appears. The larger num_workers used, the more the leak is (I ran out of 128 GB RAM in 300 iterations with batch_size of 32 and num_workers of 16). To reproduce (you should initialize images with array of filepaths to images):

import torch
import glob
from torchvision import transforms
from PIL import Image
class FramesDataset(torch.utils.data.Dataset):
    def __init__(self, images):
        self.images = images
        self.init_base_transform()

    def __len__(self):
        return len(self.images)

    def init_base_transform(self):
        self.tr_aug = transforms.Compose([transforms.GaussianBlur(7, (1, 5)),
                                          transforms.Resize((256, 256), antialias=True),
                                          transforms.ToTensor(),
                                          transforms.Normalize([0.5]*3, [0.5]*3) ])

    def __getitem__(self, idx):
        img = Image.open(self.images[idx]).convert('RGB')
        out = self.tr_aug(img)
        return out

dataset = TestDataset(images)
dl = torch.utils.data.DataLoader(dataset, batch_size = 16, num_workers = 8, pin_memory = False)
while True:
    for batch in dl:
        pass

Versions

torch: 1.12.0+cu116 torchvision: 0.13.0+cu116 PIL: 9.0.0 Ubuntu: 20.04.4 LTS

cc @vfdev-5 @datumbox

Issue Analytics

  • State:open
  • Created a year ago
  • Comments:6 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
GLivshitscommented, Aug 17, 2022

@vfdef-5 I’m just watching htop. The thing is that if your images are of the same size everything works ok (I also tried loading dataset of one image and there is no memory leak). But if there are multiple sizes of images - leak appears. It seems like there is some memory reserved for blurring operation in dataset, and if a tensor of some new size comes - leak does appear);

0reactions
GLivshitscommented, Aug 17, 2022

@vfdev-5 I’ve excluded every other augmentation and swapped the order of blur and resize and found matching leaky configuration. The code provided is already a localized version of a problem.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Memory leak in GaussianBlur · Issue #11449 - GitHub
When running cv::GaussianBlur memory is leaked. visible in the visual studio Diagnostic tools as well as the leakdump.
Read more >
Memory Leaks when Bloom or Gaussian Blur Filters used with ...
I have a score counter inside of a snapshot specifically so that I can apply the bloom filter and get a nice glow...
Read more >
opencv 3.4.2 user memory leak edit
i am using opencv 3.4.2, i wanted to know that if in an infinite for loop i keep on capturing images using a...
Read more >
Re: What is the cause of 'memory leak?: Mac Talk Forum: Digital ...
gaussian blur wrote: In Mavericks, Activity Monitor now shows memory pressure, which is a better indication of what's really happening. Would you explain...
Read more >
Raspberry pi, python, open cv memory leak? - Stack Overflow
You can pinpoint/check memory leaks yourself, using the following tools: The first tool to use is guppy/heapy - which will track all objects ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found