JSON examples for SageMaker / TF serving

cols = [
    tf.feature_column.numeric_column("age"),
    tf.feature_column.categorical_column_with_vocabulary_list("gender", ["m", "f", "other"]),
    tf.feature_column.categorical_column_with_hash_bucket("city", hash_bucket_size=15000)
]

example_spec = tf.feature_column.make_parse_example_spec

print(example_spec)
# {'gender': VarLenFeature(dtype=tf.string), 'age': FixedLenFeature(shape=(1,), dtype=tf.float32, default_value=None), 'city': VarLenFeature(dtype=tf.string)}

srv_fun = tf.estimator.export.build_parsing_serving_input_receiver_fn(example_spec)()

print(str(srv_fun))
#ServingInputReceiver(features={'city': <tensorflow.python.framework.sparse_tensor.SparseTensor object at 0x7f83c7509ad0>, 'age': <tf.Tensor 'ParseExample_7/ParseExample:6' shape=(?, 1) dtype=float32>, 'gender': <tensorflow.python.framework.sparse_tensor.SparseTensor object at 0x7f83c75093d0>}, receiver_tensors={'examples': <tf.Tensor 'input_example_tensor_8:0' shape=(?,) dtype=string>}, receiver_tensors_alternatives=None)

What format can we use to send predict requests using the Sagemaker SDK for input functions like the above?

The JSON serializer only handles arrays… so it seems like tf_estimator.predict({"city":"Paris", "gender":"m", "age":22}) is out. I tried variations of Array input and get cryptic errors from the TF serving proxy client (that source code is not available to my knowledge)

Looking at the TF Iris DNN example notebook: it uses a syntax like iris_predictor.predict([6.4, 3.2, 4.5, 1.5]) though the FeatureSpec is like {'input': IrisArrayData}. So perhaps the feature spec needs a top level?

[mvsusp]

@nkconnor,

predictor.predict should be able to handle a dictionary so tf_estimator.predict({"city":"Paris", "gender":"m", "age":22}) is the correct way to send requests like you mentioned above. The predictor logic defined here needs to handle that case.

The container source code is not open sourced yet, we are planning on releasing it very soon. I will open a bug in the container side as well, to improve the logging when the prediction fails.

I have a temporary solution to unblock you until this issue is fixed. The solution is to create a instance of a RealTimePredictor that takes a tensorflow_serving.apis request as parameter instead of a dictionary:

from sagemaker.tensorflow.predictor import tf_serializer, tf_deserializer
from tensorflow.python.saved_model.signature_constants import DEFAULT_SERVING_SIGNATURE_DEF_KEY
from tensorflow_serving.apis import classification_pb2

def _create_feature(v):
    if type(v) == int:
        return tf.train.Feature(int64_list=tf.train.Int64List(value=[v]))
    if type(v) == str:
        return tf.train.Feature(bytes_list=tf.train.BytesList(value=[v]))
    if type(v) == float:
        return tf.train.Feature(float_list=tf.train.FloatList(value=[v]))
    raise ValueError('invalid type')

endpoint = 'my-endpoint'
predictor = RealTimePredictor(endpoint=endpoint,
                              deserializer=tf_deserializer, 
                              serializer=tf_serializer,
                              content_type='application/octet-stream')

data = {'age': 39., 'workclass': 'Private', 'fnlwgt': 77516, 'education': 'Bachelors', 
        'education_num': 13.,  'marital_status': 'Never-married', 'occupation': 'Adm-clerical', 
        'relationship': 'Husband', 'race': '', 'gender': '', 'capital_gain': 2174., 'capital_loss': 0., 
        'hours_per_week': 40., 'native_country': '',  'income_bracket': '<=50K'}

features = {k: _create_feature(v) for k, v in data.items()}

request = classification_pb2.ClassificationRequest()
request.model_spec.name = "generic_model"
request.model_spec.signature_name = DEFAULT_SERVING_SIGNATURE_DEF_KEY

example = tf.train.Example(features=tf.train.Features(feature=features))

request.input.example_list.examples.extend([example])

print(predictor.predict(request))

The tensorflow_serving classification request above will be send direct to the tensorflow serving hosted inside the hosting container.

[mvsusp]

Hi @nkconnor,

Thanks for your report, I’ve confirmed that it is a bug. I’m investigating it and will give you a better update tomorrow.