Predict from in-memory dataset

Hi, currently the predict_input_fn in the demo loads the file to be scored form disc (using the specified path); however, in a real life scenario (e.g. web search) this is not realistic - the prediction must be made from a dataset already loaded into memory.

How could the demo code be modified to create a dataset on the fly from a context (query) and a set of examples (documents, which can be loaded from disc beforehand), and then use this in-memory dataset to make a prediction? Thanks.

[davidmosca]

Thanks for the tips. Does the second option (predictions = self._estimator.predict(input_fn=lambda: (features, None))) also require to rewrite the tutorial’s estimator? If so, how? And how can the features be converted into the correct format (the example shows a dictionary of arrays of floating numbers but in the Antique dataset the data (answers) are stored as protobuffers).

[xuanhuiwang]

@davidmosca, have you considered export the model to SavedModel and use the TensorFlow Serving to do the prediction?

The estimator.predict is mainly for offline analysis or debugging purpose, as far as I can tell. For production, you need to export the model and serve it outside of estimator. See https://www.tensorflow.org/tfx/tutorials/serving/rest_simple#serve_your_model_with_tensorflow_serving.