ScaNN index layer cannot be served through TF Serving docker container!

request:

curl --location --request POST 'IP_ADDRESS:8501/v1/models/retrieval:predict' \
--header 'Content-Type: application/json' \
--data-raw '{
    "instances":
        [
            "42"
        ]
}'

Response:

{
    "error": "Op type not registered 'Scann>ScannSearchBatched' in binary running on ea72279e0fdb. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed."
}

I also tried:

docker exec -it CONTAINER_ID  pip install scann

but:

ERROR: Could not find a version that satisfies the requirement scann
ERROR: No matching distribution found for scann

I think we have to copy over op source https://github.com/google-research/google-research/blob/master/scann/scann/scann_ops/py/scann_ops.py

into Serving project https://www.tensorflow.org/tfx/serving/custom_op#copy_over_op_source_into_serving_project

but I don’t know how! Can anybody help writing a custom Dockerfile?

[sammymax]

I’ll work on a ScaNN + TF Serving Dockerfile in the coming weeks–thanks for pointing this out!

[sammymax]

The Dockerfiles are ready; they’re on Docker Hub here and instructions are here. The Dockerfiles are in the linked GitHub directory, if you’re interested in seeing how they’re built.

These images were built on Clang-8 with C++17 and AVX2 enabled, while I believe the vanilla TF Serving images are built with GCC 7, C++11, and no AVX2. This means a AVX2-enabled CPU is required (any ~2013 or later x86 CPU should be ok). Otherwise, leave a comment if you run into any issues.