ImageFolder dataset: "too many open files" error

Hi, I am using ImageFolder dataset to train on imagenet. I repeatedly get “too many open files” OSError after training for several hours.
I suspect the issue comes from [here][1]:

def pil_loader(path):
    # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    with open(path, 'rb') as f:
        img = Image.open(f)
        return img.convert('RGB')  # <--

The return is inside with ... clause, thus (I suspect) the file handle f is not closed properly when the function returns. Shouldn’t the function be

def pil_loader(path):
    # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    with open(path, 'rb') as f:
        img = Image.open(f)
    return img.convert('RGB')  # <--  not indented

Thanks, -Shai [1]: https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py#L156

cc @pmeier

[vfdev-5]

@shaibagon I tested the following and no error found:

from PIL import Image

def pil_loader(path):
    # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    with open(path, 'rb') as f:
        img = Image.open(f)
        return img.convert('RGB')

import tempfile
import os

a = None

with tempfile.TemporaryDirectory() as tmp:    
    print("Temp dir: ", tmp)
    for i in range(10000000):
        if i % 50000 == 0:
            print("{}".format(i), end=" ")
        img = Image.new('RGB', size=(10, 10))
        p = os.path.join(tmp, "test_{}.png".format(i))
        img.save(p)
        a = pil_loader(p)

[shaibagon]

@vfdev-5 I am sorry, but because of the remote environment I am working at it is very difficult to provide additional information. I am also suspecting I have not located to problem correctly.

Bottom line:

running for long time (day+) using ImageFolder data results with OSError “too many open files”.

I will do my best to shed more light on this issue. meanwhile, I don’t think there’s much to do.

Other people who stumble upon this error - please try and provide more information.