Questions about YROOT

I think I miss some knowledge to configure environment variables in the following steps of “Convert the MNIST zip files into HDFS files.”

export YROOT=~/y
export LD_LIBRARY_PATH=${YROOT}/lib64:${PATH}

Specifically, how to generate ~/y/lib64 ? Any suggestions will be appreciated!

[leewyang]

I do see a problem with your command line:

In our YARN instructions, we create a Python.zip file that contains the TensorFlow dependencies. We then ship this Python.zip file to the executors, because the Hadoop nodes will typically have a different version of Python or won’t have the TensorFlow dependencies pre-installed.

When we ship the Python.zip via --archives, we also provide a “directory name” to unzip into (on the executors). In our case, we use --archives hdfs:///user/${USER}/Python.zip#Python, which means “on the executors, unzip into a directory called ‘Python’”.

We then tell the Spark executors where to find this zipped version of Python via: export SPARK_YARN_USER_ENV="PYSPARK_PYTHON=Python/bin/python". Note that the Python prefix here should match the #Python suffix of the --archives above. So, it looks like you should change your SPARK_YARN_USER_ENV.

And finally, if that still doesn’t work, please capture the yarn logs via yarn logs -applicationId <your_application_id> > yarn.log.

[leewyang]

Good to see that it worked for you, and thanks for noting your software versions. It’ll help us identify which configurations have been tried/proven.