Accuracy not as per paper

For the following code:

import torch, torchvision
from efficientnet_pytorch import EfficientNet
from tqdm import tqdm

def main(l):
    model = EfficientNet.from_pretrained(model_name='efficientnet-b'+str(l))
    model.cuda()
    model.eval()
    dataset = torchvision.datasets.ImageNet('/home/milton/.torch/datasets/', split='val', download=False,
                                                               transform=torchvision.transforms.Compose([
                                                                   torchvision.transforms.Resize(256),
                                                                   torchvision.transforms.CenterCrop(224),
                                                                   torchvision.transforms.ToTensor(),
                                                                   torchvision.transforms.Normalize((0.485, 0.456, 0.406),
                                                                                                    (0.229, 0.224, 0.225)),
                                                               ]))
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=200, shuffle=False, num_workers=40)

    with torch.no_grad():
        correct_predictions = torch.zeros(1)
        correct_topk_predictions = torch.zeros(1)
        for data, labels in tqdm(dataloader):
            data, labels = data.cuda(), labels.cuda()
            output = model(data)
            predictions = output.argmax(dim=1, keepdim=True)
            correct_predictions += predictions.eq(labels.view_as(predictions)).sum()
            _, predictions_topk = output.topk(5, 1, True, True)
            predictions_topk = predictions_topk.t()
            correct_topk_predictions += predictions_topk.eq(labels.view(1, -1).expand_as(predictions_topk)).sum()

    print(100*(correct_predictions/dataset.__len__()).item(), 100*(correct_topk_predictions/dataset.__len__()).item())


if __name__ == '__main__':
    for l in range(8):
        main(l)

I am getting the following accuracies:

Loaded pretrained weights for efficientnet-b0
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [00:57<00:00,  5.18it/s]
76.1240005493164 92.97999739646912
100%|█████████████████████████████████████████████████████████████████████████████| 31519111/31519111 [00:02<00:00, 11364441.82it/s]
Loaded pretrained weights for efficientnet-b1
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [01:10<00:00,  3.96it/s]
77.88599729537964 93.76800060272217
100%|██████████████████████████████████████████████████████████████████████████████| 36804509/36804509 [00:04<00:00, 8495839.83it/s]
Loaded pretrained weights for efficientnet-b2
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [01:13<00:00,  3.77it/s]
77.94600129127502 93.76999735832214
100%|██████████████████████████████████████████████████████████████████████████████| 49388949/49388949 [00:05<00:00, 9453856.32it/s]
Loaded pretrained weights for efficientnet-b3
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [01:31<00:00,  2.95it/s]
77.78000235557556 93.60799789428711
100%|█████████████████████████████████████████████████████████████████████████████| 77999237/77999237 [00:06<00:00, 12893608.18it/s]
Loaded pretrained weights for efficientnet-b4
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [01:57<00:00,  2.27it/s]
77.69399881362915 93.57600212097168
100%|███████████████████████████████████████████████████████████████████████████| 122410125/122410125 [00:09<00:00, 13483970.56it/s]
Loaded pretrained weights for efficientnet-b5
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [02:36<00:00,  1.68it/s]
75.95000267028809 92.69000291824341
100%|███████████████████████████████████████████████████████████████████████████| 173245619/173245619 [00:13<00:00, 12484517.39it/s]
Loaded pretrained weights for efficientnet-b6
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [03:19<00:00,  1.30it/s]
76.528000831604 93.0679976940155
Loaded pretrained weights for efficientnet-b7
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 250/250 [04:24<00:00,  1.03s/it]
76.83600187301636 93.21600198745728

The accuracies are not as per the paper. Am I doing something wrong? Please help me. Sorry for being a noob.

[drcdr]

I’m new to Efficient-Net. Using your script, I get the same results. From looking at examples/imagenet/main.py as well as #12 - image-size and interpolation are important in EfficientNets. A modified version of your script to incorporate this is:

import torch, torchvision
from efficientnet_pytorch import EfficientNet
from tqdm import tqdm
import numpy, matplotlib
import PIL

torch.backends.cudnn.benchmark = True
torch.backends.cudnn.enabled = True

def main(l):
    path_to_imagenet = YOUR-PATH-HERE
    arch = 'efficientnet-b'+str(l)
    model = EfficientNet.from_pretrained(model_name=arch)
    image_size = EfficientNet.get_image_size(arch)
    batch_size = 32  # 200 for 0, but ~32 for 7
    model.cuda()
    model.eval()
    transform = torchvision.transforms.Compose([
        torchvision.transforms.Resize(image_size, interpolation=PIL.Image.BICUBIC), # BILINEAR
        torchvision.transforms.CenterCrop(image_size),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                         std=[0.229, 0.224, 0.225]),
    ])
    dataset = torchvision.datasets.ImageNet(
        path_to_imagenet, split='val', download=False, transform=transform)
    dataloader = torch.utils.data.DataLoader(
        dataset, batch_size=batch_size, shuffle=False, num_workers=4)

    #get_model_flops(model, dataset)

    with torch.no_grad():
        correct_predictions = torch.zeros(1)
        correct_topk_predictions = torch.zeros(1)
        for data, labels in tqdm(dataloader):
            data, labels = data.cuda(), labels.cuda()
            output = model(data)
            predictions = output.argmax(dim=1, keepdim=True)
            correct_predictions += predictions.eq(labels.view_as(predictions)).sum()
            _, predictions_topk = output.topk(5, 1, True, True)
            predictions_topk = predictions_topk.t()
            correct_topk_predictions += predictions_topk.eq(
                labels.view(1, -1).expand_as(predictions_topk)).sum()

    print('Top1: %9.6f  Top5: %9.6f (imsz=%d)' %
          (100*(correct_predictions/dataset.__len__()).item(),
           100*(correct_topk_predictions/dataset.__len__()).item(), image_size))

if __name__ == '__main__':
    print_versions()
    for l in range(8):
        main(l)

See some results in the table below.

• Columns E…F are the new Tensorflow-EfficientNet AutoAugment results, just for reference
• Columns G…J are your results
• Columns K…O is the code above and should be equivalent to running examples/imagenet/main.py; scaling images using BICUBIC to a larger size.
• Columns P…T are just a variation, seeing how much worse BILINEAR is than BICUBIC.
• (1,5,d1,d5) == (Top1Val, Top5Val, Top1Val vs. Paper, Top5Val vs. Paper)

I also tried counting the model flops, using both thop and ptflops. They gave very low estimates, off by a factor of 30-50 at the high end. ptflops did get the number of params correct, just not the flops; maybe it’s because of the use of F.conv2d instead of torch.nn.Conv2d.

Here are some possible next-steps:

• convert these to ONNX, and see if the results remain the same
• make get_model_flops() work properly?
• can the new AutoAugment weights could be pulled into Efficientnet-PyTorch as well? (place to modify)

[EvgeniaAR]

@lukemelas, thanks for the quick response and this repo! We would like to compare vanilla trained efficient nets to the autoaugment ones on some tasks. If you happen to still have the pytorch vanilla weights somewhere, could you please post a download link? Otherwise, we could probably convert them from tensorflow with you code. Thanks again 😃.