1.4.4 exception: Timeout while feeding partition

Hi, I am running the mnist example with

spark-submit \
--master yarn \
--py-files ~/software/TensorFlowOnSpark-1.4.4/examples/mnist/spark/mnist_dist.py \
--conf spark.cores.max=2 \
--conf spark.task.cpus=1 \
--conf spark.executorEnv.JAVA_HOME="$JAVA_HOME" \
~/software/TensorFlowOnSpark-1.4.4/examples/mnist/spark/mnist_spark.py \
--cluster_size 2 \
--images examples/mnist/csv/train/images \
--labels examples/mnist/csv/train/labels \
--format csv \
--mode train \
--model file:///home/pi/examples/mnist/mnist_model

If I save the output in the local, it work well.

When I try to save the output to the hdfs, it will stuck and show ‘exception: Timeout while feeding partition’. How to solve it ?🙏

spark-submit \
--master yarn \
--py-files ~/software/TensorFlowOnSpark-1.4.4/examples/mnist/spark/mnist_dist.py \
--conf spark.cores.max=2 \
--conf spark.task.cpus=1 \
--conf spark.executorEnv.JAVA_HOME="$JAVA_HOME" \
~/software/TensorFlowOnSpark-1.4.4/examples/mnist/spark/mnist_spark.py \
--cluster_size 2 \
--images examples/mnist/csv/train/images \
--labels examples/mnist/csv/train/labels \
--format csv \
--mode train \
--model mnist_model

[MaQianheng]

I changed line 99 to logdir=None in mnist_dist.py. The problem was solved… 😂but, I don’t know why

[chansonzhang]

When this error occurred, I checked the container log and also find following error:

2021-04-15 16:22:33.646363: E tensorflow/core/platform/hadoop/hadoop_file_system.cc:115] HadoopFileSystem load error: libjvm.so: cannot open shared object file: No such file or directory

so I just add $LIB_JVM to --conf spark.executorEnv.LD_LIBRARY_PATH, and everything gets ok.

--conf spark.executorEnv.LD_LIBRARY_PATH=$LIB_JVM:$LIB_HDFS \