Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Flickr30k Finetune results does not match the provided checkpoint

See original GitHub issue

Hi authors,

I take the provided pretrained 200k checkpoint and did the finetuning of flickr30k. The IR and TR scores after are 64.5 and 81.7. The TR score lower than the one in the paper. My finetuning command is

$PYTHONBIN run.py with data_root=vilt_dataset/ \
        num_gpus=8 num_nodes=1 task_finetune_irtr_f30k \
        per_gpu_batchsize=4 load_path="weights/vilt_200k.ckpt" \
        exp_name="f30k/finetune_official"

I also test the given vilt_irtr_f30k.ckpt and the results is good, with IR=65.3, TR=83.5. Can I ask what is the process of getting vilt_irtr_f30k.ckpt?

Issue Analytics

State:
Created 2 years ago
Comments:10 (3 by maintainers)

Top GitHub Comments

1reaction

dandelincommented, Jun 16, 2021

@JACKHAHA363 Sure, you can grab it here https://www.dropbox.com/s/lcqmbx587szaox3/vilt_100k_wwm_pretrain.ckpt?dl=0 (will be expired someday)

0reactions

byougertcommented, Dec 31, 2021

Hi, @dandelin I’m sorry to say that the result seems still puzzled. Last night, when I changed precision to 32 during evaluation, two similar but NOT SAME results appeared, which showed one was 0.6480(ir)/0.8370(tr) but the other was 0.6460(ir)/0.8370(tr). Acatlly, seed is exactly fixed to 0. I have no idea what causes the differece. Y_Y