Exception in thread "main" java.lang.ClassNotFoundException: Could not load an Amazon Redshift JDBC driver

Trying to run the below in Intellij (without using spark-submit and adding the jdbc postgres jar file with --jars flag). I have the below basic code to test out a connection to redshift but getting the below error.

import org.apache.spark.sql._
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}
object RedshiftTest {
  def main(args: Array[String]) {
    val sparkConf = new SparkConf().setAppName("Test").setMaster("local[*]")
    val sc = new SparkContext(sparkConf)
    val sqlContext = new SQLContext(sc)
    val jdbcURL = "jdbc:redshift://<host>:5439/<db>?user=<username>&password=<pass>"
    val tempS3Dir = "s3a://<bucket>/spark_redshift_out/"
    val hadoopConf =sc.hadoopConfiguration
    hadoopConf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
    hadoopConf.set("fs.s3a.awsAccessKeyId", "<key>")
    hadoopConf.set("fs.s3a.awsSecretAccessKey", "<secret>")

    // Get some data from a Redshift table
    val df: DataFrame = sqlContext.read
      .format("com.databricks.spark.redshift")
      .option("url", jdbcURL)
      .option("dbtable", "<table>")
      .option("tempdir", tempS3Dir)
      .load()

    println(df.show())

}

Exception in thread "main" java.lang.ClassNotFoundException: Could not load an Amazon Redshift JDBC driver; see the README for instructions on downloading and configuring the official Amazon driver.
    at com.databricks.spark.redshift.JDBCWrapper$$anonfun$getDriverClass$1.apply(RedshiftJDBCWrapper.scala:80)
    at com.databricks.spark.redshift.JDBCWrapper$$anonfun$getDriverClass$1.apply(RedshiftJDBCWrapper.scala:70)
    at scala.Option.getOrElse(Option.scala:121)
    at com.databricks.spark.redshift.JDBCWrapper.getDriverClass(RedshiftJDBCWrapper.scala:69)
    at com.databricks.spark.redshift.JDBCWrapper.getConnector(RedshiftJDBCWrapper.scala:205)
    at com.databricks.spark.redshift.RedshiftRelation$$anonfun$schema$1.apply(RedshiftRelation.scala:63)
    at com.databricks.spark.redshift.RedshiftRelation$$anonfun$schema$1.apply(RedshiftRelation.scala:60)
    at scala.Option.getOrElse(Option.scala:121)
    at com.databricks.spark.redshift.RedshiftRelation.schema$lzycompute(RedshiftRelation.scala:60)
    at com.databricks.spark.redshift.RedshiftRelation.schema(RedshiftRelation.scala:59)
    at org.apache.spark.sql.execution.datasources.LogicalRelation.<init>(LogicalRelation.scala:39)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:130)
    at com.sparkexamples.RedshiftTest$.main(RedshiftTest.scala:30)
    at com.sparkexamples.RedshiftTest.main(RedshiftTest.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.ClassNotFoundException: com.amazon.redshift.jdbc4.Driver
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at com.databricks.spark.redshift.Utils$.classForName(Utils.scala:42)
    at com.databricks.spark.redshift.JDBCWrapper$$anonfun$getDriverClass$1.apply(RedshiftJDBCWrapper.scala:77)

[emptyr1]

Same issue here @JoshRosen 😦 Not sure what I’m doing wrong. I’m running spark within docker with jupyter. – the pyspark. I can access aws s3 and redshift fine without spark, so its not a permission issue. Using these commands:

import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars /home/jovyan/spark-redshift_2.10-3.0.0-preview1.jar,aws-java-sdk-1.11.534.jar,aws-java-sdk-s3-1.11.534.jar,aws-java-sdk-core-1.11.534.jar pyspark-shell'
redshift_url = "jdbc:redshift://10.200.1.205:5439/db?user=ben&password=" + urllib.parse.quote_plus("qweqwe123qweeqwe")
df = spark.read \
    .format("com.databricks.spark.redshift") \
    .option("url", redshift_url) \
    .option("query", "select * from users limit 10") \
    .option("tempdir", "/home/jovyan/redshift/") \
    .option("forward_spark_s3_credentials", "true") \
    .load()

I get error:

Py4JJavaError: An error occurred while calling o60.load.
: java.lang.ClassNotFoundException: Could not load an Amazon Redshift JDBC driver; see the README for instructions on downloading and configuring the official Amazon driver.

[nimmicv]

does this also support java ?