Low accuracy of TF-Lite model for Mobilenet (Quantization aware training)

Describe the bug The accuracy of TF-Lite model becomes extremely low after the quantization aware training of tf.keras.applications.mobilenet (v1/v2).

System information

TensorFlow installed from (source or binary): binary

TensorFlow version: tf-nightly-gpu (2.2.0.dev20200420)

TensorFlow Model Optimization version: 0.3.0

Python version: 3.6.9

Describe the expected behavior The accuracy of Keras model (with quantization aware training) and TF-Lite model are almost the same. Image classification with tools

Describe the current behavior

• Train using the tf_flowers dataset.
• Train Mobilenet V2 model without quantization aware training.
• After training, Create a quantized model using quantize_model api and train with quantization aware training.
• Check accuracy of test set with evaluate api
  • Keras model without quantization aware training: 0.99
  • Keras model with quantization aware training: 0.97
• Convert to TF-Lite model and check the accuracy with a test set. Accuracy is extremely low: 0.20%

If the model is defined as follows, the accuracy of Keras model and TF-Lite model will be almost the same.

  # extract image features by convolution and max pooling layers
  inputs = tf.keras.Input(shape = (IMG_SIZE, IMG_SIZE, 3))
  x = tf.keras.layers.Conv2D(32, kernel_size=3, padding="same", activation="relu")(inputs)
  x = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(x)
  x = tf.keras.layers.Conv2D(64, kernel_size=3, padding="same", activation="relu")(x)
  x = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(x)
  # classify the class by fully-connected layers
  x = tf.keras.layers.Flatten()(x)
  x = tf.keras.layers.Dense(512, activation="relu")(x)
  x = tf.keras.layers.Dense(info.features['label'].num_classes)(x)
  x = tf.keras.layers.Activation("softmax")(x)
  model_functional = tf.keras.Model(inputs=inputs, outputs=x)

Code to reproduce the issue (Google Colab notebook) https://gist.github.com/NobuoTsukamoto/b42128104531a7612e5c85e246cb2dac

Screenshots

Additional context

[nutsiepully]

We’ve found the issue. One of the quantized kernel activation ranges had a problem, but was getting hidden when the range has converged.

We’ll have a fix out soon. tf-nightly should have it.

[nutsiepully]

Thanks a lot @sayakpaul. Really appreciate the feedback and the effort.

Thanks @kmkolasinski and @NobuoTsukamoto for the detailed bug reports and feedback. I’m closing the bug. Please reopen if you face any further issues.

@sayakpaul, the report is awesome! Great work, this explains the value of the tooling really well.