Do we need to implement `_prepare_split`?

Describe the bug

I’m not sure this is a bug or if it’s just missing in the documentation, or i’m not doing something correctly, but I’m subclassing DatasetBuilder and getting the following error because on the DatasetBuilder class the _prepare_split method is abstract (as are the others we are required to implement, hence the genesis of my question):

Traceback (most recent call last):
  File "/home/jason/source/python/prism_machine_learning/examples/create_hf_datasets.py", line 28, in <module>
    dataset_builder.download_and_prepare()
  File "/home/jason/.virtualenvs/pml/lib/python3.8/site-packages/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/jason/.virtualenvs/pml/lib/python3.8/site-packages/datasets/builder.py", line 793, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/jason/.virtualenvs/pml/lib/python3.8/site-packages/datasets/builder.py", line 1124, in _prepare_split
    raise NotImplementedError()
NotImplementedError

Steps to reproduce the bug

I will share implementation if it turns out that everything should be working (i.e. we only need to implement those 3 methods the docs mention), but I don’t want to distract from the original question.

Expected behavior

I just need to know if there are additional methods we need to implement when subclassing DatasetBuilder besides what the documentation specifies -> _info, _split_generators and _generate_examples

Environment info

• datasets version: 2.4.0
• Platform: Linux-5.4.0-135-generic-x86_64-with-glibc2.2.5
• Python version: 3.8.12
• PyArrow version: 7.0.0
• Pandas version: 1.4.1

[mariosasko]

the requirement of a loading script has always seemed counterintuitive to me

This is a requirement only for datasets not stored in standard formats such as CSV, JSON, SQL, Parquet, ImageFolder, etc.

if i have to provide a script with every dataset, what is the point of using datasets if we’re doing all the work of loading it, I can just do that in my code and skip the datasets integration (this of course discounts other potential benefits around metadata management, etc., my example is just simplest use case though for the sake of discussion)

Our README/documentation lists the main features…

One of the main ones is that our library makes it easy to work with datasets larger than RAM (thanks to Arrow and the caching mechanism), and this is not trivial to implement.

Regarding the step-by-step builder, this is the pattern:

from datasets import load_dataset_builder
builder = load_dataset_builder("path/to/script") # or direct instantiation with MyDatasetBuilder(...)
builder.download_and_prepare()
dset = builder.as_dataset()

[mariosasko]

cc @stevhliu who may have some ideas on how to improve this part of the docs.