Correct way of exporting keras model

Bug Description

I have faced a bug around exporting and keras model to .h5 file, loading into another keras code, and using the model. It is similar to issue #167 .

After searching, fitting, final fitting a model using Auto-Keras, I exported the model as a keras model file. Then, I loaded the model into another “pure keras” code. However, the performance of the loaded model was so much poorer than the one evaluated by an Auto-Keras code.

Reproducing Steps

I have run the sample code on the official website:

from keras.datasets import mnist
from autokeras.image_supervised import ImageClassifier

if __name__ == '__main__':
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train = x_train.reshape(x_train.shape + (1,))
    x_test = x_test.reshape(x_test.shape + (1,))

    clf = ImageClassifier(verbose=True)
    clf.fit(x_train, y_train, time_limit=12 * 60 * 60)
    clf.final_fit(x_train, y_train, x_test, y_test, retrain=True)
    y = clf.evaluate(x_test, y_test)
    print(y)

fit and final_fit finished, and the print(y) outputted 0.9926, the accuracy score. After the code written above, I added one-line code for exporting the keras model:

clf.load_searcher().load_best_model().produce_keras_model().save('my_model.h5')

Next, I loaded the model my_model.h5 into another script and evaluated again with keras evaluate function.

from keras.datasets import mnist
from keras.models import load_model
from keras.utils import to_categorical

if __name__ == '__main__':
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_test = x_test.reshape(x_test.shape + (1,))
    y_test = to_categorical(y_test)

    keras_model = load_model('my_model.h5')
    keras_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
    
    score = keras_model.evaluate(x_test, y_test)
    print(score)

In this code, I added compile function because the error message was shown:

/home/morningyrp/tensorflow/venv/local/lib/python3.5/site-packages/keras/engine/saving.py:269: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  warnings.warn('No training configuration found in save file: '

The print(score) outputted [4.088045687866211, 0.1313], which means that the accuracy score was 0.1313. In spite of the same MNIST data, this result is so much poorer the result from clf.evaluate in Auto-Keras, 0.9926.

Finally, I tried to train the model again with keras codes like this:

history = keras_model.fit(x_train, y_train,\
                    batch_size=32,\
                    epochs=1000,\
                    verbose=1,\
                    validation_split=0.1)

However, the accuracy could not increase and the loss could not reduce through whole epochs.

I also tried to export the best model without final_fit of Auto-Keras, loaded it to another keras code, and trained using compile and fit function of pure keras. However, the accuracy and the loss could not improve through whole epochs.

Expected Behavior

The keras model exported by clf.load_searcher().load_best_model().produce_keras_model().save('my_model.h5') can be used correctly on another “pure keras” code. In other words, fit, evaluate, predict functions of the exported model can be used correctly in other keras codes.

Setup Details

Include the details about the versions of:

• OS type and version: Linux (Ubuntu 16.04)
• Python: 3.5.2
• autokeras: 0.2.13
• scikit-learn:
• numpy: 1.14.5
• keras: 2.2.2
• scipy: 1.1.0
• tensorflow: 1.10.1
• pytorch: 0.4.1

[AShyshkov]

You are we welcome! (I have to say, the fix was found not by me, but by my manager)

The code is exactly what we did. Just FYI, you can do this instead: keras_model.layers[:-1].activation = keras.activations.softmax

The result would be the same.

[AShyshkov]

Hey, I think we found the issue. Seem like a defect in AutoKeras. The last layer in the best model for MNIST classifier is a Dense layer with linear activation function. Training such model ends up with exploding gradients issue. Change it to softmax (or top it off with layer Activation(‘softmax’) ), retrain your model and you should be fine.