Loss cannot drop

Thank you so much for sharing your codes. I try to employ Vit as the encoder and follow a common decoder to build a segmentation network. I train it from scratch but found the loss can’t drop since the beginning of training, and the results keep near 0. Is there any trick for training Vit correctly? Is it very important to load the pre-train model to fine-tune? Here is my configuration: patch_size=16 hidden_size=16*16*3 mlp_dim = 3072 dropout_rate = 0.1 num_heads = 12 num_layers = 12 lr=3e-4 opt=Adam weight_decay=0.0

[QiushiYang]

@lucidrains OK. Thanks a lot for your kind suggestions 😃

[QiushiYang]

Thanks a lot for your helps. I will try to remove the conv-based decoder in the segmentation pipeline. Besides, I have another question that can original learnable position embedding (with random initialization) learn the spatial information well in spatial-sensitive tasks such as semantic segmentation?