Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

CLEVR pretrained model gives FID 22

See original GitHub issue

Hi, kudos for great work!

I’ve just noticed that with recommended preprocessing and evaluation, the metrics on gdrive:cityscapes work as expected (FID ~5.2), while for CLEVR exactly two same lines:

python prepare_data.py --clevr --max-images 100000
python run_network.py --eval --gpus 0 --expname clevr-exp --dataset clevr --pretrained-pkl gdrive:clevr-snapshot.pkl

give ~22 FID, not 9.2. Can you please double-check if the provided snapshot is correct? Or am I missing smth here?

Thanks in advance!

Issue Analytics

State:
Created 2 years ago
Comments:8 (6 by maintainers)

Top GitHub Comments

1reaction

doraradcommented, Apr 27, 2021

Hi, thank you very much for pointing that out! I’ll check that today. I verified all scores locally so there must be a bug / incorrect snapshot file. Will get back shortly!

0reactions

doraradcommented, May 2, 2021

Alright I uploaded the data (the difference between the locally processed data that leads to the FID score 12 is that the images ratio height/weight in the data is 2/3 while the pretrained model was trained on images with ratio 3/4. You can try again to run python prepare_data.py --clevr --max-images 100000 should take a few minutes only (because will just download the data) and then another 20min to run the python run_network.py --eval --gpus 0 --expname clevr-exp --dataset clevr --pretrained-pkl gdrive:clevr-snapshot.pkl . Let me know if you’re still getting higher FID!

Top Results From Across the Web

ashkamath/mdetr

This repository contains code and links to pre-trained models for MDETR (Modulated DETR) for pre-training on data having aligned text and images with...

deep learning - Which is the fastest image pretrained model?

The answer will depend on some things such as your hardware and the image you process. Additional, we should distinguish if you are...

CLEVR-X: A Visual Reasoning Dataset for Natural ...

The CLEVR dataset [27] was proposed as a diagnostic dataset to inspect the visual reasoning of VQA models. Multiple frameworks have been ...

Image Recognition with Transfer Learning (98.5%)

Use transfer learning to easily classify dog and cat pictures with a 98.5% accuracy.

CLEVR: A Diagnostic Dataset for Compositional Language ...

We use CLEVR to analyze a suite of VQA models and discover weaknesses that are not widely known. For exam- ple, we find...