Can't run on v3-8 or v3-32 TPU nodes.

Hi, I trained TPU-accelerated GANs from https://github.com/tensorflow/gan without any issues, but can’t seem to get compare_gan examples to run on GCP TPUs.

Here is the general error, which appears whether using ctpu, gcloud, or the online GUI to setup compute resources.

tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation input_pipeline_task0/TensorSliceDataset: node input_pipeline_task0/TensorSliceDataset (defined at /usr/local/lib/python3.5/dist-packages/tensorflow_core/python/framework/ops.py:1748) was explicitly assigned to /job:worker/task:0/device:CPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0 ]. Make sure the device specification refers to a valid device.

Any thoughts here? Is there a specific python/tensorflow version I should use for running compare_gan?

Thanks!

[mbbrodie]

You’re welcome. And yes, you’ll need to pip install googleapiclient and oauth2client as well. For simplicity, you could also change cp /usr/local/cuda-10.1/lib64/libcudart.so libcudart.so.10.0 (and other cp commands) to sudo cp /usr/local/cuda-10.1/lib64/libcudart.so /usr/local/cuda-10.1/lib64/libcudart.so.10.0.

Otherwise, make sure you put the renamed libs on your LD_LIBRARY_PATH.

[hytseng0509]

@mbbrodie Thanks for the information! Do you encounter the error message that request the installation of googleapiclient and oauth2client in your setup?