CLIPTextModel gives invalid output for zeroed attention mask

System Info

• transformers version: 4.21.2
• Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.33
• Python version: 3.8.6
• Huggingface_hub version: 0.5.1
• PyTorch version (GPU?): 1.12.0+cu102 (False)
• Tensorflow version (GPU?): 2.6.0 (False)
• Flax version (CPU?/GPU?/TPU?): 0.4.2 (cpu)
• Jax version: 0.3.10
• JaxLib version: 0.3.10
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help?

@patil-suraj

Reproduction

from transformers import CLIPTextModel
import torch

model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32")

inputs = {
  "input_ids": torch.tensor([[49406, 320, 1125, 539, 320, 1929, 49407]]),
  "attention_mask": torch.tensor([[0, 0, 0, 0, 0, 0, 0]])
}

outputs = model(**inputs)

Given the zeroed attention mask, the attention weights should be all equal here:

https://github.com/huggingface/transformers/blob/21f6f58721dd9154357576be6de54eefef1f1818/src/transformers/models/clip/modeling_clip.py#L246

However, causal and attention masks are added separately (here), so in this case, before going through softmax, certain values are twice as small as the other ones (to be more precise, some values are -min_float and other are -inf). Consequently, softmax outputs probabilities that match the causal mask.

This is also the case for TFCLIPTextModel.

[jonatanklosko]

So I think we can close this 😃

The additive mask before softmax is a trick that works under the assumption that the mask has at least a single 1. So from what I understand can be said that for zeroed mask the output of most models is just not well defined, and it’s fine given the use cases so far.

[patil-suraj]

Thanks a lot for the issue @jonatanklosko !

This indeed seems a bit strange, I see two solutions here

• We could combine causal mask and attention mask and see what happens
• Or instead of using additive masks, we could replace the masked values with large negative numbers.

however, is it really a bug ? as long as the values for masked positions are much lower than non-masked tokens, those tokens will still be ignored. Do you have an example where you see the masked positions are not ignored ?

@seanmor5 good point!

I’ve also experienced this issue, and anecdotally this inconsistency seems to impact the quality of stable diffusion outputs from https://github.com/huggingface/diffusers

I’m not sure if this impacts the quality of diffusers, for example, as discussed in this issue, we have verified that the results are 1:1 with the original repo.

I am assuming Stable Diffusion was trained using the PT CLIPTextModel, and thus results rely on this inconsistent/invalid text embedding?

Yes, it was trained using CLIPTextModel, but both this training and the actual pre-trained CLIP model never used attention mask. They always pad the sequence to max_len 77 and use causal mask. This is how we recommend to use the stable diffusion model. I know this is not ideal but that’s how it was trained. cf https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/encoders/modules.py#L155

Also cc @patrickvonplaten , wdyt ?