Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

jupyter-scala with Spark does not see imports

See original GitHub issue

I am running jupyter notebook on the machine where is available configured spark distribution with spark-submit , spark-defaults etc. I have running Python and R kernels but I have issues with jupyter-scala. I try to run the following code inside the notebook;

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import com.datastax.spark.connector._

val conf = new SparkConf(true)
val sc = new SparkContext("spark://127.0.0.1:7077", "test", conf)

but it seems the jars are not visible by it

Main.scala:29: object datastax is not a member of package com
 ; import org.apache.spark.SparkConf ; import org.apache.spark.SparkContext ; import com.datastax.spark.connector._ ; val conf = { () =>
                                                                                         ^
Main.scala:29: object apache is not a member of package org
 ; import org.apache.spark.SparkConf ; import org.apache.spark.SparkContext ; import com.datastax.spark.connector._ ; val conf = { () =>
              ^
Main.scala:30: not found: type SparkConf
new SparkConf(true) 
    ^
Main.scala:29: object apache is not a member of package org
 ; import org.apache.spark.SparkConf ; import org.apache.spark.SparkContext ; import com.datastax.spark.connector._ ; val conf = { () =>
                                                  ^
Main.scala:33: not found: type SparkContext
new SparkContext("spark://127.0.0.1:7077", "test", conf) 
    ^

I have tried to add before it

classpath.addPath("/var/lib/spark/lib/spark-assembly-1.6.0-hadoop2.6.0.jar")
classpath.add("datastax" % "spark-cassandra-connector" % "1.6.0-M1-s_2.11")

but nothing changes.

I am running jupyter-scala from https://git.io/vzhRi .

Issue Analytics

State:
Created 8 years ago
Comments:5 (2 by maintainers)

Top GitHub Comments

1reaction

drizhamcommented, Mar 16, 2016

This is how I went about solving the issue I had (similar to you) Installed Jupyter Toree from here: https://github.com/apache/incubator-toree

0reactions

dimon777commented, Mar 29, 2018

I run into the same issue with Spark 2.3 and Scala 2.11. I used this repo to integrate with jupyter but seeing same error. Is there Toree agnostic way to integrate Spark+Scala and Jupyter? Also spark assembly is no longer presented in spark distribution. Perhaps your README and This notebook has to be updated

Top Results From Across the Web

apache spark - Import custom scala object in jupyter notebook ...

1 Answer 1 · --jars option on spark commandline · spark.sparkContext.addJar(/path/to/JAR/file).

How To Use Jupyter Notebooks with Apache Spark

In this post, we will see how to incorporate Jupyter Notebooks with ... Scala is the ideal language to interact with Apache Spark...

Databricks Connect - Azure - Microsoft Learn

With Databricks Connect, you can: Run large-scale Spark jobs from any Python, Java, Scala, or R application. Anywhere you can import pyspark , ......

PixieDust 3 - Scala and Python - | notebook.community

Unfortunately, Jupyter Python notebooks do not currently provide a way to call out Scala or Java code. As a result, a typical workaround...

almond-sh/almond - Gitter

Noting that ammonite-spark did not like the latest Ammonite version, ... See https://almond.sh/docs/api-jupyter.html#updatable-display-data. plotly-scala ...