Different prediction results after loaded model training

I implemented the Deep autoencoder from https://blog.keras.io/building-autoencoders-in-keras.html and I am currently trying to save the trained model and restore it to enhance it later.

The problem: When I run the example with 36 epochs in 1 step I am getting different results as when I run the example with 12 epochs in 3 steps.

1 Step with 36 epochs:

3 Steps á 12 epochs:

Shouldn’t it be the same result or am I mistaken here? Thank you in advance for any clarification.

This is the current code:

# Taken from https://blog.keras.io/building-autoencoders-in-keras.html
import os.path

from keras.layers import Input, Dense
from keras.models import Model, load_model

# model file name
model_file_name = 'mnist-autoencoder.h5'

# encoder file name
encoder_file_name = 'mnist-autoencoder-encoder.h5'

# decoder file name
decoder_file_name = 'mnist-autoencoder-decoder.h5'

# this is the size of our encoded representations
encoding_dim = 32  # 32 floats -> compression of factor 24.5, assuming the input is 784 floats

# this is our input placeholder
input_img = Input(shape=(784,))

# "encoded" is the encoded representation of the input
encoded = Dense(128, activation='relu')(input_img)
encoded = Dense(64, activation='relu')(encoded)
encoded = Dense(32, activation='relu')(encoded)

# "decoded" is the lossy reconstruction of the input
decoded = Dense(64, activation='relu')(encoded)
decoded = Dense(128, activation='relu')(decoded)
decoded = Dense(784, activation='sigmoid')(decoded)

# create a placeholder for an encoded (32-dimensional) input
encoded_input = Input(shape=(encoding_dim,))

# load or create model
if os.path.isfile(model_file_name) and os.path.isfile(encoder_file_name) and os.path.isfile(decoder_file_name):
    autoencoder = load_model(model_file_name)
    encoder = load_model(encoder_file_name)
    decoder = load_model(decoder_file_name)

else:
    # this model maps an input to its encoded representation
    encoder = Model(input=input_img, output=encoded)

    # this model maps an input to its reconstruction
    autoencoder = Model(input=input_img, output=decoded)

    # First, we'll configure our model to use a per-pixel binary crossentropy loss,
    # and the Adadelta optimizer:
    autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

    # retrieve decoder layers of autoencoder model
    decoder_layer1 = autoencoder.layers[-3]
    decoder_layer2 = autoencoder.layers[-2]
    decoder_layer3 = autoencoder.layers[-1]

    # create the decoder model
    decoder = Model(input=encoded_input, output=decoder_layer3(decoder_layer2(decoder_layer1(encoded_input))))

# Training

# Let's prepare our input data. We're using MNIST digits, and we're discarding
# the labels (since we're only interested in encoding/decoding the input images).
from keras.datasets import mnist
import numpy as np
(x_train, _), (x_test, _) = mnist.load_data()

# We will normalize all values between 0 and 1 and we will flatten the
# 28x28 images into vectors of size 784.
x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
print x_train.shape
print x_test.shape

# Now let's train our autoencoder for 50 epochs:
autoencoder.fit(
    x_train,
    x_train,
    nb_epoch=12,
    batch_size=256,
    shuffle=True,
    validation_data=(x_test, x_test)
)

# Save trained autoencoder model
autoencoder.save(model_file_name, overwrite=True)

# Save encoder with weights
encoder.save(encoder_file_name, overwrite=True)

# Save decoder
decoder.save(decoder_file_name, overwrite=True)

# After 50 epochs, the autoencoder seems to reach a stable train/test loss
# value of about 0.11. We can try to visualize the reconstructed inputs and
# the encoded representations. We will use Matplotlib.
#
# encode and decode some digits
# note that we take them from the *test* set
encoded_imgs = encoder.predict(x_test)
decoded_imgs = decoder.predict(encoded_imgs)

# use Matplotlib (don't ask)
import matplotlib.pyplot as plt

n = 10  # how many digits we will display
plt.figure(figsize=(20, 4))
for i in range(n):
    # display original
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(x_test[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # display reconstruction
    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(decoded_imgs[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

plt.show()

[fbrix]

@bstriner Thank you, that was it! I am reconstructing the layers now as you mentioned and it works.

[bstriner]

@fbrix Just re-read your code and realized what the issue probably is. Saving and reloading breaks the relationship between autoencoder/encoder/decoder. So after reloading, training the autoencoder no longer updates the encoder. You are probably seeing the autoencoder training as normal but the decoder and encoder not updating.

You should be able to verify by checking if your encoder and autoencoder have the same weights for the first layer before and after training.

1. You can try just saving the autoencoder. Only load the autoencoder. Then you can recreate the encoder and decoder by something like this after loading the autoencoder (indexes are probably wrong but you get the idea):

x = autoencoder.inputs
h = autoencoder.layers[1](x)
h = autoencoder.layers[2](h)
encoder = Model(x, h)

1. You can create the encoder and decoder as separate models. Create autoencoder=Model(x, decoder(encoder(x))). Save and load just the autoencoder. The encoder and decoder should be autoencoder.layers[0] and autoencoder.layers[1] (or maybe 1 and 2, have to test). Similar to the above solution but makes it easier to pull the encoder and decoder out of the autoencoder.

2. Use save_weights and load_weights. That does not create new objects but loads weights into existing objects. This way it won’t break the shared weights but you won’t get to save the optimizer state.

Cheers, Ben