How to speed up the data loader?

❓ Questions and Help

I followed the Tutorials to use custom datasets(DeepFashion 2). But I found it very slow when run trainer = DefaultTrainer(cfg). It seems that it takes much time in loading datas, maybe 20-30mins. I changed cfg.DATALOADER.NUM_WORKERS from 1 to 4 to 12, it didn’t help(my cpu have 12 cores), results shows even worse. So how to speed up the data loader? I write a script to test loading data with multi-thread, finding it only takes 4-5mins. Is it pytorch problems?

[ppwwyyxx]

The original code that loads COCO dataset is fast enough for training builtin models.

If you found the dataloader slow for your dataset, then the reason could be in your dataset, your custom dataloader, your machine. Without any details provided about what you did or what you observed, we cannot give a valid response. So closing.

[invisprints]

Maybe you read image matrix in custom datasets. In my case, I just write image file name in custom instead of image itself. It just loads slow but costs little memory. And I think the reasons why it runs slowly is that there’re too many small config files. I write a script to convert my datasets to coco format config (just one file). Now it run much faster than before.