Error when using generated FastSpeech2 TFLite model with Android example

Hi, this one’s probably for @mapledxf and is a duplicate of this issue: https://github.com/TensorSpeech/TensorFlowTTS/issues/274

I’ve converted my FastSpeech2 model to a TFLite model in the way shown on Colab.

I’ve taken the output TFLite model, put it in the assets folder on the Android Example, and renamed it to fastspeech2_quant.tflite to replace the existing fastspeech2 tflite. Then when I type in some text into the box and press speak, an error occurs.

Here’s the Android app logs:

Android App error logs

D/AudioTrack: ClientUid 10397 AudioTrack::start 
D/TtsStateDispatcher: onTtsReady: 
I/AssistStructure: Flattened final assist data: 3320 bytes, containing 1 windows, 14 views
D/InputWorker: add to queue: testing testing this is a voice test
D/InputWorker: processing: testing testing this is a voice test
D/TtsStateDispatcher: onTtsStart: 
D/InputWorker: speak: [testing testing this is a voice test]
D/processor: text preprocessed: testing testing this is a voice test
D/FastSpeech2: input id length: 36
E/InputWorker: Exception: 
    java.lang.IllegalArgumentException: Cannot convert between a TensorFlowLite tensor with type FLOAT32 and a Java object of type [I (which is compatible with the TensorFlowLite type INT32).
        at org.tensorflow.lite.Tensor.throwIfTypeIsIncompatible(Tensor.java:427)
        at org.tensorflow.lite.Tensor.getInputShapeIfDifferent(Tensor.java:287)
        at org.tensorflow.lite.NativeInterpreterWrapper.run(NativeInterpreterWrapper.java:146)
        at org.tensorflow.lite.Interpreter.runForMultipleInputsOutputs(Interpreter.java:360)
        at com.tensorspeech.tensorflowtts.module.FastSpeech2.getMelSpectrogram(FastSpeech2.java:67)
        at com.tensorspeech.tensorflowtts.tts.InputWorker$InputText.proceed(InputWorker.java:86)
        at com.tensorspeech.tensorflowtts.tts.InputWorker$InputText.access$700(InputWorker.java:66)
        at com.tensorspeech.tensorflowtts.tts.InputWorker.lambda$new$0$InputWorker(InputWorker.java:43)
        at com.tensorspeech.tensorflowtts.tts.-$$Lambda$InputWorker$jYcArgm2l9qkL5ylkeKfv_zVAG8.run(Unknown Source:2)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:919)

Here’s some suspicious logs made when doing the conversion, which may be related:

TFLite Conversion Logs

2020-10-30 23:25:24.454753: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._0/attention/self/Shape that is not type float.
2020-10-30 23:25:24.470916: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._0/attention/self/Shape that is not type float.
2020-10-30 23:25:24.488655: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._0/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.509365: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._0/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.533000: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._1/attention/self/Shape that is not type float.
2020-10-30 23:25:24.553011: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._1/attention/self/Shape that is not type float.
2020-10-30 23:25:24.573864: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._1/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.595381: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._1/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.619833: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._2/attention/self/Shape that is not type float.
2020-10-30 23:25:24.638449: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._2/attention/self/Shape that is not type float.
2020-10-30 23:25:24.657566: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._2/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.677951: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._2/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.698034: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._3/attention/self/Shape that is not type float.
2020-10-30 23:25:24.717519: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._3/attention/self/Shape that is not type float.
2020-10-30 23:25:24.738637: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._3/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.758332: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor encoder/layer_._3/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.778157: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor duration_predictor/dense_3/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.803053: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor duration_predictor/dense_3/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.822459: I tensorflow/lite/tools/optimize/quantize_weights.cc:211] Skipping quantization of tensor duration_predictor/dense_3/Tensordot/MatMul because it has fewer than 1024 elements (256).
2020-10-30 23:25:24.844940: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor energy_predictor/dense_2/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.868953: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor energy_predictor/dense_2/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.888406: I tensorflow/lite/tools/optimize/quantize_weights.cc:211] Skipping quantization of tensor energy_predictor/dense_2/Tensordot/MatMul because it has fewer than 1024 elements (256).
2020-10-30 23:25:24.909410: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor f0_predictor/dense_1/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.936395: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor f0_predictor/dense_1/Tensordot/Shape that is not type float.
2020-10-30 23:25:24.958783: I tensorflow/lite/tools/optimize/quantize_weights.cc:211] Skipping quantization of tensor f0_predictor/dense_1/Tensordot/MatMul because it has fewer than 1024 elements (256).
2020-10-30 23:25:24.990119: I tensorflow/lite/tools/optimize/quantize_weights.cc:211] Skipping quantization of tensor length_regulator/Repeat/boolean_mask/Reshape because it has fewer than 1024 elements (1).
2020-10-30 23:25:25.020595: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._0/attention/self/Shape that is not type float.
2020-10-30 23:25:25.043419: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._0/attention/self/Shape that is not type float.
2020-10-30 23:25:25.065443: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._0/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:25.090323: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._0/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:25.113152: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._1/attention/self/Shape that is not type float.
2020-10-30 23:25:25.134870: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._1/attention/self/Shape that is not type float.
2020-10-30 23:25:25.155454: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._1/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:25.178455: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._1/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:25.210206: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._2/attention/self/Shape that is not type float.
2020-10-30 23:25:25.240226: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._2/attention/self/Shape that is not type float.
2020-10-30 23:25:25.265304: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._2/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:25.289809: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._2/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:25.317303: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._3/attention/self/Shape that is not type float.
2020-10-30 23:25:25.343394: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._3/attention/self/Shape that is not type float.
2020-10-30 23:25:25.376316: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._3/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:25.405087: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor decoder/layer_._3/attention/output/dense/Tensordot/Shape that is not type float.
2020-10-30 23:25:25.426959: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor mel_before/Tensordot/Shape that is not type float.
2020-10-30 23:25:25.449968: I tensorflow/lite/tools/optimize/quantize_weights.cc:203] Skipping quantization of tensor mel_before/Tensordot/Shape that is not type float.
Writing converted model to: .\model-215000.tflite

Any ideas what’s going on here? Thanks 😃

[mapledxf]

according to this issue , new version model has removed attention mask as an input. so u should also delete attention mask input for android which is coded here https://github.com/TensorSpeech/TensorFlowTTS/blob/e42595abbf21208c81e0fabaa0b1eaeaca2c4053/examples/android/app/src/main/java/com/tensorspeech/tensorflowtts/module/FastSpeech2.java#L68

[mapledxf]

@OscarVanL yeah, ur right. That’s why i didnt make the PR to correct this.