Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Bug in Masked Patch Prediction ?

See original GitHub issue

Hi,

It seems that the label generation for Masked Patch Prediction are wrong. The loss converges to 0 after a couple of iterations as the generated labels remain constant and the same in all and every iterations. Here’s an example of generated labels:

tensor([292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292, 292,
        292, 292, 292, 292])

I appreciate if you can look into this.

Issue Analytics

State:
Created 2 years ago
Comments:12 (9 by maintainers)

Top GitHub Comments

2reactions

Siyuan89commented, Apr 27, 2021

Thank you @zankner . Just for reference, reproducing can be done using this snippet:

import torch
from vit_pytorch_source import ViT
from vit_pytorch_source.mpp import MPP

model = ViT(
    image_size=256,
    patch_size=32,
    num_classes=1000,
    dim=1024,
    depth=6,
    heads=8,
    mlp_dim=2048,
    dropout=0.1,
    emb_dropout=0.1
)

mpp_trainer = MPP(
    transformer=model,
    patch_size=32,
    dim=1024,
    mask_prob=0.15,          # probability of using token in masked prediction task
    random_patch_prob=0.30,  # probability of randomly replacing a token being used for mpp
    replace_prob=0.50,       # probability of replacing a token being used for mpp with the mask token
)

opt = torch.optim.Adam(mpp_trainer.parameters(), lr=3e-4)

def sample_unlabelled_images():
    return torch.randn(20, 3, 256, 256)


for _ in range(100):
    images = sample_unlabelled_images()
    loss = mpp_trainer(images)
    print(loss)
    opt.zero_grad()
    loss.backward()
    opt.step()

2reactions

zanknercommented, Apr 27, 2021

@lucidrains @Siyuan89 Thanks for reporting this. I’ll take a look at it today.

Top Results From Across the Web

Who is Sir Bug A Boo? 'The Masked Singer' Prediction & Clues!

Ken predicted it was Jordan Peele based on the clues. Nicole guessed that it could be Lil Rel, who she did Drop the...

Ray Parker Jr. ('The Masked Singer' Sir Bug a Boo) unmasked ...

The entire panel consisting of Robin Thicke, Jenny McCarthy, Ken Jeong and Nicole Scherzinger failed to correctly guess Grammy-winning swinger- ...

Masked Singer 2022 updates — All season 8's remaining ...

Masked Singer 2022 updates — All season 8's remaining clues and predictions as Scarecrow and Bugaboo identities revealed. Who is Snowstorm?

Generating Bug-Fixes Using Pretrained Transformers

We frame bug-patching as a sequence-to-sequence learning task consisting of ... as with adding an auxiliary task to predict each token's syntactic class....

An Empirical Study on Learning Bug-Fixing Patches in the ...

Overall, this model is capable of predicting fixed patches generated by developers in 9--50% of the cases, depending on the number of candidate...