IAM Role not taken into account

Hi,

I launch a job with spark-shell or spark-submit:

spark-shell --name "my test app" --packages com.databricks:spark-redshift_2.11:2.0.1

then

sparkSession.read.schema(schema).
      format("com.databricks.spark.redshift").
      option("url", redshiftUrl).
      option("user",redshiftUser).
      option("password",redshiftPassword).
      option("aws_iam_role", redshiftRoleArn).
      option("query", query).
      option("tempdir", s3TmpDir).
      load()

and I get the exception:

java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified
as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId
 or fs.s3n.awsSecretAccessKey properties
at com.databricks.spark.redshift.S3Credentials.initialize(S3Credentials.java:67)
at com.databricks.spark.redshift.AWSCredentialsUtils$.com$databricks$spark$redshift$AWSCredentialsUtils$$loadFromURI(AWSCredentialsUtils.scala:60)
    at com.databricks.spark.redshift.AWSCredentialsUtils$$anonfun$load$1.apply(AWSCredentialsUtils.scala:48)
    at com.databricks.spark.redshift.AWSCredentialsUtils$$anonfun$load$1.apply(AWSCredentialsUtils.scala:48)
    at scala.Option.getOrElse(Option.scala:121)
    at com.databricks.spark.redshift.AWSCredentialsUtils$.load(AWSCredentialsUtils.scala:48)
    at com.databricks.spark.redshift.RedshiftRelation.buildScan(RedshiftRelation.scala:89)
    at org.apache.spark.sql.execution.datasources.DataSourceStrategy$$anonfun$8.apply(DataSourceStrategy.scala:260)
    at org.apache.spark.sql.execution.datasources.DataSourceStrategy$$anonfun$8.apply(DataSourceStrategy.scala:260)
    at org.apache.spark.sql.execution.datasources.DataSourceStrategy$$anonfun$pruneFilterProject$1.apply(DataSourceStrategy.scala:303)
...

The role works using psql client and UNLOAD write the result on s3TmpDir. I checked the code, and I am not really sure how IAM role can be handled… It looks like a bug to me.

PS: using spark-submit, my app have the following dependencies in build.sbt

   libraryDependencies ++= Seq(
      "com.databricks" %% "spark-redshift" % "2.0.1" % Compile,
      "joda-time" % "joda-time" % "2.9.4" % Compile,
      "com.amazonaws" % "aws-java-sdk-core" % "1.11.37" % Compile,
      "com.amazonaws" % "aws-java-sdk-s3" % "1.11.37" % Compile,
      "org.scalaj" %% "scalaj-http" % "2.3.0" % Compile,
      "net.liftweb" %% "lift-json" % "2.6.3" % Compile,
      "com.amazon.redshift" % "jdbc42.Driver" % "1.1.17.1017" from "https://s3.amazonaws.com/redshift-downloads/drivers/RedshiftJDBC42-1.1.17.1017.jar",
      "org.apache.spark" %% "spark-core" % "2.0.0" % Compile,
      "org.apache.spark" %% "spark-sql" % "2.0.0" % Compile,
      "org.apache.spark" %% "spark-hive" % "2.0.0" % Compile,
      "org.scalatest" %% "scalatest" % "3.0.0" % Test
    )

[JoshRosen]

The problem here relates to Redshift <-> S3 communication, not Spark <-> S3.

I think that you’ll have to either use the AWS security token service instructions (mentioned at https://github.com/databricks/spark-redshift/issues/252#issuecomment-237910447) or will need to use the aws_iam_role parameter to tell Redshift which role to assume when communicating with S3.

I’ve been meaning to write a tutorial on using this library with only IAM authentication because I think that the current featureset makes it possible but not as easy at it could be.

[alinden]

I encountered this issue with code identical to the original post. The problem seems to be that instance based authentication is not supported for s3://... or s3n://... urls. Changing to s3a://... resolved the issue.