Issue of reusing epoch_iter introduced from the latest commit

@ngoyal2707 @myleott At this line from this commit, self.epoch_iter will not be updated if not None, i.e., the iterator would be the same across epochs. However, we need different seeds for the epoch iterator in different epochs for shuffling; otherwise we will get the same batch order across epochs… So IMO returning same iterator across epochs might not be a good idea in any cases.

PS: This also made the validation log printing starting from epoch 2. I think it is because the training iterator will even be reused for validation when reusing self.epoch_iter like that.

[ngoyal2707]

yeah, the reason to change this was similar to what you mentioned as we were seeing slightly different results between single and multi gpu setup because of this extra rand() call in multi-gpu case.

I don’t think the right solution here is to keep the rand() call. DO you see big difference in your experiments with and without it? Worse case, I’d suggest add that rand() call in your task __init__?

(cc: @myleott )

[freewym]

@ngoyal2707 OK, I find the reason why my previous experiment was not reproducible: this commit replaced torch.rand() with torch.zeros(), affecting the state of RNG when doing multi-gpu training. It may sound strange, but is it possible to revert back to torch.rand() so that all the previous experiments are reproducible?