Inference failed, ValueError: Shapes (149, 512) and (199, 512) are incompatible

I am using batch size 2 in tacotron2.v1.yaml with a custom 22050 Hz, mono, 10h total myDataset. I prepared it and normalized it. I started

python examples/tacotron2/train_tacotron2.py --train-dir ./dump_myDataset/train/ --dev-dir ./dump_myDataset/valid/ --outdir ./examples/tacotron2/exp/train.tacotron2.v1/ --config ./examples/tacotron2/conf/tacotron2.v1.yaml --use-norm 1 --mixed_precision 0 --resume ""

and waited for a few checkpoints.

I am trying to use the standard python script for inference with my checkpoint model-2073.h5 and the pretrained melgan model. I am using the ljspeech_mapper.json from the dump folder of my model.

I get

Traceback (most recent call last):
  File "standardpythonscript.py", line 15, in <module>
    pretrained_path="./examples/tacotron2/exp/train.tacotron2.v1/checkpoints/model-2073.h5"
  File "E:\tts\inference\auto_model.py", line 69, in from_pretrained
    model.load_weights(pretrained_path)
  File "E:\Anaconda\envs\myEnv\lib\site-packages\tensorflow\python\keras\engine\training.py", line 2211, in load_weights
    hdf5_format.load_weights_from_hdf5_group(f, self.layers)
  File "E:\Anaconda\envs\myEnv\lib\site-packages\tensorflow\python\keras\saving\hdf5_format.py", line 708, in load_weights_from_hdf5_group
    K.batch_set_value(weight_value_tuples)
  File "E:\Anaconda\envs\myEnv\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "E:\Anaconda\envs\myEnv\lib\site-packages\tensorflow\python\keras\backend.py", line 3576, in batch_set_value
    x.assign(np.asarray(value, dtype=dtype(x)))
  File "E:\Anaconda\envs\myEnv\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py", line 858, in assign
    self._shape.assert_is_compatible_with(value_tensor.shape)
  File "E:\Anaconda\envs\myEnv\lib\site-packages\tensorflow\python\framework\tensor_shape.py", line 1134, in assert_is_compatible_with
    raise ValueError("Shapes %s and %s are incompatible" % (self, other))
ValueError: Shapes (149, 512) and (199, 512) are incompatible

import numpy as np
import soundfile as sf
import yaml

import tensorflow as tf

from tensorflow_tts.inference import AutoConfig
from tensorflow_tts.inference import TFAutoModel
from tensorflow_tts.inference import AutoProcessor

# initialize tacotron2 model.
fs_config = AutoConfig.from_pretrained('./examples/tacotron2/conf/tacotron2.v1.yaml')
tacotron2 = TFAutoModel.from_pretrained(
    config=fs_config,
    pretrained_path="./examples/tacotron2/exp/train.tacotron2.v1/checkpoints/model-2073.h5"
)


# initialize melgan model
melgan_config = AutoConfig.from_pretrained('./examples/melgan/conf/melgan.v1.yaml')
melgan = TFAutoModel.from_pretrained(
    config=melgan_config,
    pretrained_path="./examples/melgan/checkpoint/generator-1500000.h5"
)


# inference
processor = AutoProcessor.from_pretrained(pretrained_path="./dump_myDataset/ljspeech_mapper.json")

ids = processor.text_to_sequence("my german testing sentence")
ids = tf.expand_dims(ids, 0)
# tacotron2 inference

masked_mel_before, masked_mel_after, duration_outputs = tacotron2.inference(
    ids,
    speaker_ids=tf.zeros(shape=[tf.shape(ids)[0]], dtype=tf.int32),
    speed_ratios=tf.constant([1.0], dtype=tf.float32)
)

# melgan inference
audio_before = melgan.inference(masked_mel_before)[0, :, 0]
audio_after = melgan.inference(masked_mel_after)[0, :, 0]

# save to file
sf.write('./audio_before.wav', audio_before, 22050, "PCM_16")
sf.write('./audio_after.wav', audio_after, 22050, "PCM_16")

Ideas?

[dathudeptrai]

@ErfolgreichCharismatisch can you share ur mapper.json ?. 149 is the number of ljspeech charactor, but seem ur model have 199 charactor/phoneme input. Did you change ur config file ?, please see here (https://github.com/TensorSpeech/TensorFlowTTS/blob/master/examples/tacotron2/conf/tacotron2.v1.yaml#L19). you should write ur own processor and ur symbols set then add ur vocab_size here (https://github.com/TensorSpeech/TensorFlowTTS/blob/master/tensorflow_tts/configs/tacotron2.py#L59-L68)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.