A project ID is required

Hi,

I try to use the spark connector on my Apache Spark cluster with Scala but can’t because of the following error: java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment. Please set a project ID using the builder.

Here my code

import org.apache.spark.sql.SparkSession
import com.google.cloud.spark.bigquery._

val spark = SparkSession.builder
  .getOrCreate()

spark.conf.set("credentialsFile", "my.json")

spark.read.bigquery("project_id.datasets.tablel")

I installed spark_bigquery_latest.jar on the cluster and even try the global environment variable option but nothing works.

[davidrabinowitz]

Hi @morganvill

Is the credential file a personal one or a service account? Personal credentials files do not contain the project id. You can either set the parent project id in the code in the read option: spark.read.format("bigquery").option("parentProject", "<your-gcp-project-id>").option("table", "project_id.datasets.tablel").load(); Alternatively you can set the GOOGLE_CLOUD_PROJECT environment variable, pointing to your project id.

[Paulito123]

Adding the option .option(“parentProject”, “<your-gcp-project-id>”) did the trick for me. Thanks for the help!