ranking installation silently upgrades the TF version.

I’m using TF 2.4.1 version that is compatible with the Horovod version I have in place. When I install the ranking library, it silently upgrades the TF to the latest version that later conflicts with the Horovod. Following is the stack strace.

[1,10]<stderr>:Traceback (most recent call last):
[1,10]<stderr>: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-a988a17e-1421-48f9-b928-65cd81681867/lib/python3.8/site-packages/horovod/tensorflow/mpi_ops.py", line 47, in <module>
[1,10]<stderr>: MPI_LIB = _load_library('mpi_lib' + get_ext_suffix())
[1,10]<stderr>: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-a988a17e-1421-48f9-b928-65cd81681867/lib/python3.8/site-packages/horovod/tensorflow/mpi_ops.py", line 42, in _load_library
[1,10]<stderr>: library = load_library.load_op_library(filename)
[1,10]<stderr>: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-a988a17e-1421-48f9-b928-65cd81681867/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py", line 58, in load_op_library
[1,10]<stderr>: lib_handle = py_tf.TF_LoadLibrary(library_filename)
[1,10]<stderr>:tensorflow.python.framework.errors_impl.NotFoundError: /local_disk0/.ephemeral_nfs/envs/pythonEnv-a988a17e-1421-48f9-b928-65cd81681867/lib/python3.8/site-packages/horovod/tensorflow/mpi_lib.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN10tensorflow11GetNodeAttrERKNS_9AttrSliceEN4absl14lts_2020_02_2511string_viewEPf
[1,10]<stderr>:
[1,10]<stderr>:The above exception was the direct cause of the following exception:
[1,10]<stderr>:
[1,10]<stderr>:Traceback (most recent call last):
[1,10]<stderr>: File "<string>", line 1, in <module>
[1,10]<stderr>: File "/databricks/.python_edge_libs/sparkdl/horovod/runner.py", line 217, in wrapped_main
[1,10]<stderr>: return_value = main(**kwargs)
[1,10]<stderr>: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-a988a17e-1421-48f9-b928-65cd81681867/lib/python3.8/site-packages/lodgingsort/runner.py", line 103, in _run_multi_worker_training
[1,10]<stderr>: import horovod.tensorflow as hvd
[1,10]<stderr>: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-a988a17e-1421-48f9-b928-65cd81681867/lib/python3.8/site-packages/horovod/tensorflow/__init__.py", line 26, in <module>
[1,10]<stderr>: from horovod.tensorflow import elastic
[1,10]<stderr>: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-a988a17e-1421-48f9-b928-65cd81681867/lib/python3.8/site-packages/horovod/tensorflow/elastic.py", line 24, in <module>
[1,10]<stderr>: from horovod.tensorflow.functions import broadcast_object, broadcast_object_fn, broadcast_variables
[1,10]<stderr>: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-a988a17e-1421-48f9-b928-65cd81681867/lib/python3.8/site-packages/horovod/tensorflow/functions.py", line 24, in <module>
[1,10]<stderr>: from horovod.tensorflow.mpi_ops import allgather, broadcast
[1,10]<stderr>: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-a988a17e-1421-48f9-b928-65cd81681867/lib/python3.8/site-packages/horovod/tensorflow/mpi_ops.py", line 49, in <module>
[1,10]<stderr>: check_installed_version('tensorflow', tf.__version__, e)
[1,10]<stderr>: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-a988a17e-1421-48f9-b928-65cd81681867/lib/python3.8/site-packages/horovod/common/util.py", line 268, in check_installed_version
[1,10]<stderr>: raise HorovodVersionMismatchError(name, version, installed_version) from exception
[1,10]<stderr>:horovod.common.exceptions.HorovodVersionMismatchError: Framework tensorflow installed with version 2.4.1 but found version 2.5.0.
[1,10]<stderr>: This can result in unexpected behavior including runtime errors.
[1,10]<stderr>: Reinstall Horovod using `pip install --no-cache-dir` to build with the new version.

I see in the setup file of the ranking project, we check for ‘tensorflow-serving-api >= 2.0.0, < 3.0.0’, ‘tf-models-official >= 2.5.0’ packages and install the latest one. I tried modifying this to 2.4.0 version but it did not help as installing the 2.4 version of tensorflow-serving-api and tf-models-official also upgrading the TF version to 2.5.

It would be a great help if you can suggest a way to use the ranking package with TF 2.4.1 version? Thanks in advance.

[xuanhuiwang]

I guess the reason is that you don’t have tensorflow-serving-api installed before. When this is the case, the latest version of tensorflow-serving-api is then installed, which also upgrades the tensorflow package.

What if you manually install the tensorflow-serving-api to the same version as the tensorflow version? Then try to install the tf-ranking v0.3.3?

[motiteux]

for our use-case (same as @DbVikas in Databricks), we do not use TL serving api, since we use tf ranking for training. What is our solution for this?