The code of OFA-base is inconsistent with the pre-trained checkpoint

Thanks for your awesome work. Something has bothered me recently. When I continued to train OFA-base (I tried to collect all the pre-training data of OFA), I found that a few training steps (10 steps) would make the performance of OFA-base worse. I checked the config in checkpoint and the config in pretrain_ofa_base.sh, and found many differences. What might affect the results?

In addition, I found that there is a dimension inconsistency in the network. decoder.image_position_idx": "<class 'torch.Tensor'> torch.Size([1026]) in code and decoder.image_position_idx": "<class 'torch.Tensor'> torch.Size([1025]) in ckpt. Is this the reason for the decline of performance?

[logicwong]

@zzhanghub @Charles-Xie Changing the dimension of image_position_idx to 1026 is to be consistent with the dimension of embed_positions, it shouldn’t affect the performance. I guess you save the model during training, but even if the model parameters are not updated, the parameters of batch_norm will still be updated and degenerate the performance. You can set --freeze-resnet to fix the problem.

@zzhanghub For detailed differents between resaved ckpt and ofa-base ckpt:

1. That’s ok.
2. In order to improve the pre-training, we use a larger sample_patch_num in the later stage of training. But in recent experiments, we found that using --patch-image-size=256 without sampling patches is better.
3. As mentioned above, this does not affect performance。
4. The path of neg_sample_dir has some sensitive information, so we manually deleted it before releasing ckpt.
5. The model was interrupted many times during pre-training, we reset the learning rate a few times during this period, but this should have little effect on the performance

[zzhanghub]

Thank you !!!