[RFC] Wrapper/Proxy for Stream

🚀 Feature

Discussed with @NivekT about the wrapper class for all streams: Pros:

• We can add a __del__ method to close the file stream automatically when ref count becomes 0 for wrapper. It would eliminate all warnings.
• A wrapper class can unify the reading API for file streams. (For OnDiskCache, I would prefer a unified API to read stream, otherwise I have to handle all different cases)
  • Local file stream, we can use read() to read everything into memory
  • When we set stream=True for large file, the requests.Response doesn’t support read. It only supports iter_content or __iter__ to read chunk by chunk.

Cons:

• Thanks to @NivekT, it needs extra care about magic methods.

Reference: https://github.com/pytorch/data/pull/35#discussion_r728201731, https://github.com/pytorch/data/pull/65#discussion_r730117933

cc: @VitalyFedyunin

[ejguan]

In the case of Extractor or Decompressor the yielded file handle should also be wrapped by the Wrapper to make sure fd is closed automatically and have a unified API like read

[ejguan]

You are right with Saver. I am talking about reading data from file stream. The current workflow is:

urls = IterableWrapper([URL])
fd = urls.open_url() # File handles
data = fd.map(fn=lambda x: x.read(), input_col=1) # <- This one is referred as Downloader in my mind
file = data.save_to_disk()

We can let user to do any map function to download data from file handle. But, if we are going to implement a DataPipe to do the same thing, we need to make sure all file handles (streams) sent to this DataPipe can be read.

The reason that I prefer a read method is the stream type varies:

• String: ".join(fd)"
• Bytes: b"".join(fd)