Predefined ImageNets `predict` method is inaccurate

Hello. I have been using Chainer predefined ImageNets (chainer.links.VGG16Layers etc.) but recently found out that they are inaccurate than one can usually expect.

Specifically, one can basically use either __call__ or predict to run inference and I think the result should be the same if an equivalent preprocessing is appropriately conducted. However in my quick experiments the results are significantly different. With VGG and GoogLeNet, __call__ performs better whereas predict works better with ResNets.

Here is Top-5 accuracy of single-image (no crop, no oversample, no ensemble) inference (I used only 2000 samples)

CNN	__call__	predict
GoogLeNet	86.75%	74.1%
VGG16Layers	85.2%	74.0%
ResNet50Layers	12.7%	78.6%
ResNet101Layers	15.1%	78.75%
ResNet152Layers	14.8%	80.65%

I’m not pretty sure if it’s a chainer-side problem or just I’m doing wrong, but as far as I checked the document (and briefly read the source code) I couldn’t find that I’m doing badly.

Here is the reproduction code and the result. https://gist.github.com/belltailjp/7677987c699b2f3cf766ecc89958dff2 Run prepare.sh followed by python run.py. You can change CNN type by just comment out, and switch __call__ and predict by simply changing use_predict = False line. I confirmed this issue with the following environment: Chainer 3.0.0, Cupy 2.0.0, Ubuntu 16.04, CUDA8.0, cuDNN7, Python 3.6.1

[belltailjp]

@ronekko It solved!!! It’s also a common pitfall… (of course I tried both RGB&BGR, but I changed the color order for both predict and __call__, that’s why I couldn’t notice…) This should definitely be properly documented. I will submit a small document update patch. Thanks so much @chenzhekl , too

[stale[bot]]

This issue is closed as announced. Feel free to re-open it if needed.