Low eval accuracy for TextClassifier (Might because of cannot save text_vectorization)

I have run the Text Classification example here: https://autokeras.com/tutorial/text_classification/

My validation accuracy gets to the high 90s, but when I run inference on the test set, it is close to 50%. There’s nothing inherently wrong with that if that’s the expected output. However, it looks as though it is just using the last model trained and not the best model based on the printed results from the last model being trained.

Also, when I run clf.evaluate(x_train, y_train) I get the same results of 50% accuracy. I apologize if I’m missing something obvious here, but this does not look right to me.

[haifeng-jin]

@dzimmerman-nci Thank you for the work solution! Hope tf would release this “fit to adapt” feature soon.

[dzimmerman-nci]

@haifeng-jin it seems the only way to get it to work is if I retrain the exported Keras model after adapting the TextVectorization layer like this:

auto_keras_model = ak.TextClassifier(max_trials=3)
auto_keras_model.fit(x_train, y_train)
keras_model = auto_keras_model.export_model()
text_vectorization_layer = keras_model.layers[1]
text_vectorization_layer.adapt(x_train)
keras_model.fit(x_train, y_train)
print(keras_model.evaluate(x_test, y_test))

If I exclude the keras_model.fit(x_train, y_train) line, I get the same results as I do when I do not adapt the TextVectorization layer.