Multiprocessing with any DataPipe writing to local file

🐛 Describe the bug

We need to take extra care all DataPipe that would write to file system when DataLoader2 triggered multiprocessing. If the file name on the local file system is same across multiple processes, it would be a racing condition. This is found when TorchText team is using on_disk_cache to cache file. DataLoader needs to know such DataPipe must be sharded with multiprocessing or enforce it into single process.

As a workaround, users have to download the file to local file system to prevent writing within DataPipe.

Versions

main branch

[NivekT]

A mutex shared across processes would be sufficient in my mind. But, I don’t know if there is a cross-platform solution for it.

@hudeven is looking into the solution in this diff D35459528. Basically it is using io_path’s file locking mechanism which internally depends on portalocker. would this be a viable cross-platform solution?

I think if we can incorporate that into the IoPathSaver DataPipe, it should be a viable cross-platform solution, but it would mean users have to install iopath and portalocker if they wish to lock files across processes.

[NicolasHug]

Thanks for the ping Parmeet. I haven’t encountered this issue thus far, because torchvision datasets do not write anything to disk.