On masked-lm labels and computing the loss

Recently I was using bert for my own project, and going through the function mask_tokens I found this line of code labels[~masked_indices] = -100 # We only compute loss on masked tokens I wonder why we do this? like i get the part where we do

    indices_replaced = torch.bernoulli(torch.full(labels.shape, 0.8)).bool() & masked_indices
    inputs[indices_replaced] = tokenizer.convert_tokens_to_ids(tokenizer.mask_token)

To mask the input tokens but is it necessary for labels? Like if I had a constant -100 as ground truth and the actual id maybe say 1000 the loss may never converge

And I’ve found two contradictory comments ie labels[~masked_indices] = -100 # We only compute loss on masked tokens and

```(run_language_modeling)
masked_lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
            Labels for computing the masked language modeling loss.
            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
            Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
            in ``[0, ..., config.vocab_size]
```(modeling_bert)

One says loss will be computed on masked and another says will be ignored… Could anyone please let me know about it… Thanks.

[gmaratos]

@LysandreJik Isn’t the example mentioned in the official documentation missing the following line of code before feeding labels into model?

labels[inputs.input_ids != tokenizer.mask_token_id] = -100

I believe, with this we calculate the negative log likelihood, just for the masked token which is `Paris’ in the given example.

Yes, I was wondering why this is missing as well. There doesn’t seem to be any documentation indicating that this is happening automatically before the loss is computed. And, based on some limited testing on my end I get different values for the loss when I do this.

[achingacham]

@LysandreJik Isn’t the example mentioned in the official documentation missing the following line of code before feeding labels into model?

labels[inputs.input_ids != tokenizer.mask_token_id] = -100

I believe, with this we calculate the negative log likelihood, just for the masked token which is `Paris’ in the given example.