help with training

Thanks for the awesome code! I am training my own model right now and have a few questions:

• currently I am using 100k (out of around 1.8m) images from CelebAMask-HQ, ffhq and vggface to train the model. did you use the full set to train your model?
• I didn’t see large improvement for most losses anymore (160k steps trained, 4gpus x 12images/batch); is this normal? should I just continue training for more steps?
• I also checked the validation results, and the reconstruction is not good.
• I noticed shuffle for the training dataloader is not set to True, did you use the same setting?

Thanks!

[niuyuanc]

@antonsanchez Could you please share the pretrained weights? Thank you so much 🙏🙏🙏 My email: [e-mail] Thanks🙏🙏🙏

[y-x-c]

Hi! You did very fast training!

1. Yes, I used full-set dataset. I don’t know about IJB-C dataset. The distribution of dataset can influence to your model.
2. In the paper, they trained for 500K steps. I trained for over 500K. In my eye, your losses are getting down for attribute_loss but unstable for Rec and ID loss. In my case, the two losses are more stable and lower at the same steps.
3. shuffle option in training dataloader should be True. It is clearly my mistake while publishing.

Thanks for your reply.

1. I just corrected the description, I am using the same datasets (CelebAMask-HQ, ffhq and vggface) as well.

• So in your case, each step has 64 images; and let’s say there are 1.5m images in those three datasets, so you trained for around 4 epochs (= 64 * 500000 / 1500000 / 5 ) in total?
• In my case, each step only has 48 images, so maybe that’s why the two losses are higher at the same steps.
• I found the Rec loss is going much lower in the third epoch, and the results are much better than before. I will continue my current training and see what’s going on.

1. Thanks for the clarification, I also changed to True during my training.