How to save tokenize data when training from scratch

❓ Questions & Help

I am training Allbert from scratch following the blog post by hugging face. As it mentions that :

If your dataset is very large, you can opt to load and tokenize examples on the fly, rather than as a preprocessing step.

How this can be done any suggestions? As of now , using the method given in the notebook:

from transformers import TextDataset

dataset = TextDataset(
    tokenizer=tokenizer,
    file_path="./oscar.eo.txt",
    block_size=128,
)

there is no method to save tokenize data, can anyone suggest how to save that as its already taking long enough before starting training.

[008karan]

That worked @LysandreJik thanks! I still not getting how you can prepare pretraining data on the fly while training. I got large training data and don’t want to wait until it gets prepared for training.

[LysandreJik]

Once you have your data you can pickle it or use torch.save to save it to your disk and reload it later.