Can't import sparkdl with spark-deep-learning-assembly-0.1.0-spark2.1.jar

First of all, thank you for a great library!

I tried to use sparkdl in PySpark, but couldn’t import sparkdl. Detailed procedure is as follows:

# make sparkdl jar
build/sbt assembly

# run pyspark with sparkdl
pyspark --master local[4] --jars target/scala-2.11/spark-deep-learning-assembly-0.1.0-spark2.1.jar

# import sparkdl
import sparkdl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named sparkdl

After digging a few places, I found that it works if I deflate the jar file as follows.

cd target/scala-2.11
mkdir tmp
cp spark-deep-learning-assembly-0.1.0-spark2.1.jar tmp/
cd tmp
jar xf spark-deep-learning-assembly-0.1.0-spark2.1.jar

pyspark --jars spark-deep-learning-assembly-0.1.0-spark2.1.jar

import sparkdl
Using TensorFlow backend.

Edited-1 : The second method works only in the directory where the jar file is deflated.

Best wishes, HanCheol

[priancho]

After a few hours of googling/testing, I finally found a complete solution for my problem.

When I run pyspark in local mode, --packages option is enough to import sparkdl.

$ pyspark --master local[1] --packages databricks:spark-deep-learning:0.1.0-spark2.1-s_2.11

>>> import sparkdl
Using TensorFlow backend.

But it breaks when pyspark runs in yarn mode.

$ pyspark --master yarn --packages databricks:spark-deep-learning:0.1.0-spark2.1-s_2.11

>>> import sparkdl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named sparkdl

Interestingly, spark-submit runs a python program having the same import code without any problem.

Based on the information found here, https://issues.apache.org/jira/browse/SPARK-5185, I manually added paths to the downloaded jars into sys.path variable (which is equivalent to PYTHONPATH). And it started to work.

$ pyspark --master yarn --packages databricks:spark-deep-learning:0.1.0-spark2.1-s_2.11

>>> import sys, glob, os
>>> sys.path.extend(glob.glob(os.path.join(os.path.expanduser("~"), ".ivy2/jars/*.jar")))
>>>
>>> import sparkdl
Using TensorFlow backend.
>>> my_images = sparkdl.readImages("data/flower_photos/daisy/*.jpg")
>>> my_images.show()
+--------------------+--------------------+
|            filePath|               image|
+--------------------+--------------------+
|hdfs://mycluster/...|[RGB,263,320,3,[B...|
|hdfs://mycluster/...|[RGB,313,500,3,[B...|
|hdfs://mycluster/...|[RGB,215,320,3,[B...|
...

Based on the results of these try&error, it seems like pyspark in yarn mode doesn’t properly set PYTHONPATH for the jar files added by --packages option.

Best wishes, Hancheol

[sueann]

If you don’t need to change the code, the easiest way to start pyspark with the library would be

pyspark --packages databricks:spark-deep-learning:0.1.0-spark2.1-s_2.11

If you need to use the jar, it should work if you also add the jar to the PYTHONPATH environment variable before the pyspark command. e.g. for bash:

export PYTHONPATH=$PYTHONPATH:$<path/to/assembly/jar> 

Hope that works!