How to split `_RawTextIterableDataset`

❓ Questions and Help

I am trying to move from using legacy and use new provided features, i was doing this:

from torchtext import legacy
TEXT = legacy.data.Field(lower=True, batch_first=True)
LABEL = legacy.data.LabelField(dtype=torch.float)
train_data, test_data = legacy.datasets.IMDB.splits(TEXT, LABEL, root='/tmp/imdb/')
train_data, valid_data = train_data.split(split_ratio=0.8, random_state=random.seed(SEED))

But now i want to split train_data, how can i do that?

from torchtext.datasets import IMDB
train_iter, test_iter = IMDB(split=('train', 'test'))
# I need to split train_iter into train_iter and valid_iter

And i think providing more features more than just this one would help more, Thanks!

[zhangguanheng66]

It’s an iterator so I don’t think you can split/shuffle it. I think, it’s worth an option to set up the offset or the beginning of line. So for the valid set, you can start from a different line. cc @cpuhrsch @parmeet

[ejguan]

it depends on how you want to split. For a simple case, you can use demux to split based on the indices generated by enumerating from the prior DataPipe.