Clarification about installation with spark

Hi,

I am trying to install the GCS connector. The documentation state:

Note that you do not need to configure Hadoop in order to use the GCS connector with Spark.

But I am getting the following exception:

java.io.IOException: No FileSystem for scheme: gs
  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
  at org.apache.spark.sql.execution.datasources.DataSource$.org$apache$spark$sql$execution$datasources$DataSource$$checkA

[medb]

Yes, this is a shaded jar.

Shaded jar is a library jar that includes all its dependencies, it has -shaded suffix in the name in the Maven repo and when you are building project: https://repo1.maven.org/maven2/com/google/cloud/bigdataoss/gcs-connector/hadoop3-2.0.0/

[hadrienk]

Ok, I got it working:

"spark.hadoop.fs.gs.impl": {
    "value": "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS",
    "type": "string"
},
"spark.hadoop.google.cloud.auth.service.account.enable": {
    "value": true,
    "type": "checkbox"
},
"spark.hadoop.google.cloud.auth.service.account.json.keyfile": {
    "value": "/zeppelin/key.json",
    "type": "string"
}

The PR is using a newer version maybe? https://github.com/GoogleCloudPlatform/bigdata-interop/pull/180/files because com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem was not found in the jar from the documentation.