Train on large textfile

Hi, I’m trying to train a model from scratch as I want it to generate text in another language (Swedish). My trainingdata is a large collection of novels, about 22 000 that are all in one single .txt-file delimited by a line with only a <s> The txt-file is about 300MB in size. However, both when I try to train it from scratch using the Colab notebook(with a P100 GPU) or locally on my desktop it runs out of memory and crashes. My desktop has 32GB RAM and a Geforce 2080Ti with 11GB VRAM.

Is there any way to make aitextgen work with 300MB trainingdata? Are there any parameters I can tweak to have it use less memory? Should I arrange the trainingdata in another way?

[mohataher]

@minimaxir, so this thread discusses pretty much the same thing. I have a large file, 1.5 GB specifically of Arabic text that I want to train on.

Is there a way this library could handle a file this size? Like training on one batch at a time or split the file into chunks and feed the trainer gradually?

[zephyo]

Update: Got it working by setting batch_size=128, num_workers=2, and running:

python
import torch
torch.cuda.empty_cache()