Requesting some more training details

Hey again, just wanted to ask for a bit more training detail. I’m trying to reproduce results (computing benchmark metrics on the Leaderboard validation routes), and the pretrained weights seem to do better than the weights I train on my own setup. I have two questions about how main_model_10.th is trained

1. How large was the main_dataset used to train that model?
2. How long did it take to train the model? I’m assuming it was 10 epochs total, which would’ve taken multiple days for the 100 Gb main dataset I was using

Also, thank you for the reply on the other issue - I’ll need to mull over your answers and read through the code some more to understand what’s going on. I’ll reply to that issue once I’ve thought about it some more!

[dotchen]

How large was the main_dataset used to train that model?

So the main_dataset we used to train our leaderboard model is the one that we released in DATASET.md. It consists of roughly one million frames. This is also mentioned in our paper. See Section 5 in the paper.

How long did it take to train the model? I’m assuming it was 10 epochs total, which would’ve taken multiple days for the 100 Gb main dataset I was using

10 Epochs. It took me 4 days to train on two titan Xp with the default batch size. 100GB sounds a bit small, the dataset I used (RAILS_1M) before compression is 3.4TB in the lmdb format.

[dotchen]

This error usually happens when you have a preceding error, please check if it throws any other error before it.

P.S. this is a separate issue (together with others in this thread) than indicated in the title. Please consider open a separate thread so it would be easier for others to navigate. Thanks!