Reading in multiple files in one go.

Is your feature request related to a problem? Please describe.

I have a folder with lots and lots of json files. Can I read them in all at once?

Describe the solution you’d like

It’d be nice if all of our readers allowed for a '*' or a Path.glob that allows you to read in lots of files at once.

Something like:

Clumper.read_json("folder/*.json") 
Clumper.read_json(pathlib.Path("folder").glob("*"))

Additional context

As far as an implementation goes, we can probably solve this nicely with a decorator. Assuming the function that it wraps is a file-reader we should not need to touch the internal readers.

[samarpan-rai]

@koaning Gefeliciteerd!

I agree with @samukweku. It makes things more complicated. I would suggest we support the following inputs for reading jsonl

# Using string
Clumper.read_jsonl("folder/*.jsonl")
Clumper.read_jsonl("folder/*.jsonl", n=50) # 50 lines per found file

# Using Path.glob
Clumper.read_jsonl(pathlib.Path("folder").glob("*.jsonl"))
Clumper.read_jsonl(pathlib.Path("folder").glob("*.jsonl"), n=50) # 50 lines per found file

I am having some trouble with accessing the path argument in the decorator, maybe we can discuss how to solve it? We could discuss it in this issue but it would be unnecessarily bloated with messages so I suggest we talk in some chat platform. Is there any platform you prefer to talk in? I can create a room in gitter for these discussions as it has easy integration with github.

[samukweku]

I dont think we should. separating single and multiple files seems to be an easier and more maintainable approach