Inference error: shape mismatch

Hi, I was testing live_asr.py on my macOS Monterey (Python=3.8.11) under the following environment:

halo==0.0.31
numpy==1.21.4
PyAudio==0.2.11
Rx==3.2.0
SoundFile==0.10.3.post1
torch==1.10.0
torchaudio==0.10.0
transformers==4.8.2
webrtcvad==2.0.10

When I run python live_asr.py, I came across errors as below:

Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-large-960h-lv60-self and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

listening to your voice

/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/transformers/feature_extraction_utils.py:158: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
  tensor = as_tensor(value)
/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py:986: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  return (input_length - kernel_size) // stride + 1
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/jkang/Desktop/test_asr/wav2vec2-live/live_asr.py", line 92, in asr_process
    text = wave2vec_asr.buffer_to_text(float64_buffer).lower()
  File "/Users/jkang/Desktop/test_asr/wav2vec2-live/wav2vec2_inference.py", line 22, in buffer_to_text
    logits = self.model(inputs.input_values, attention_mask=torch.ones(len(inputs.input_values[0]))).logits
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1120, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1528, in forward
    return_dict=return_dict,
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1120, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1171, in forward
    return_dict=return_dict,
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 791, in forward
    hidden_states[~attention_mask] = 0
IndexError: The shape of the mask [1920, 5] at index 0 does not match the shape of the indexed tensor [1, 5, 1024] at index 0

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/jkang/Desktop/test_asr/wav2vec2-live/live_asr.py", line 68, in vad_process
    frame = stream.read(CHUNK)
  File "/Users/jkang/anaconda3/envs/jk/lib/python3.7/site-packages/pyaudio.py", line 608, in read
    return pa.read_stream(self._stream, num_frames, exception_on_overflow)
OSError: [Errno -9981] Input overflowed

I think the critical issue is this:

IndexError: The shape of the mask [1920, 5] at index 0 does not match the shape of the indexed tensor [1, 5, 1024] at index 0

I installed transformers==4.8.2, but the error occurs which is probably not related to the transformers’ version in my guess.

Could you help me with what caused this error?

Thank you

[jaekookang]

Yes, it works! Thank you for taking your time to fix it @oliverguhr.

[oliverguhr]

Great - you are welcome.