Reading BigQuery table in PySpark outside GCP

My server runs on AWS. I follow the instructions here and this tutorial script, but I get Py4JJavaError caused by missing project ID

Caused by: java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment.  Please set a project ID using the builder.
	at com.google.cloud.spark.bigquery.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
	at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:266)
	at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryOptions.<init>(BigQueryOptions.java:81)
	at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryOptions.<init>(BigQueryOptions.java:30)
	at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryOptions$Builder.build(BigQueryOptions.java:76)
	at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryOptions.getDefaultInstance(BigQueryOptions.java:136)
	at com.google.cloud.spark.bigquery.BigQueryRelationProvider$.$lessinit$greater$default$2(BigQueryRelationProvider.scala:30)
	at com.google.cloud.spark.bigquery.BigQueryRelationProvider.<init>(BigQueryRelationProvider.scala:40)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	... 24 more

My python script looks like this:

from pyspark.sql import SparkSession
spark = (
    SparkSession.builder
    .appName('bq')
    .master('local[4]')
    .config('spark.jars', '/path/to/bigquery_spark-bigquery-latest.jar')
    .getOrCreate()
)
spark.conf.set("credentialsFile", "/path/to/credentials.json")

df = (
    spark.read
    .format('bigquery')
    .option('project', 'myProject')
    .option('table', 'myTable')
    .load()
)

Any idea how could I fix the missing project ID error?

[xjrk58]

Hi @FurcyPin,

I would recommend specifying parent (billing) project option as well (.option("parentProject", "billing-project-id")).

[moscich]

@KhanhTrinh1703 Thanks for a quick response!

I think that suddenly I figured out whats going on. After adding parentProject option I was authenticated to bigquery but not into google cloud storage.

I needed to pass my key also into hadoopConfiguration:

spark.sparkContext._jsc.hadoopConfiguration().set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
spark.sparkContext._jsc.hadoopConfiguration().set("fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")
spark.sparkContext._jsc.hadoopConfiguration().set("google.cloud.auth.service.account.enable", "true")
spark.sparkContext._jsc.hadoopConfiguration().set("google.cloud.auth.service.account.json.keyfile", "/Users/marekmoscichowski/Documents/Dev/pyspark/key2.json")

I only wonder if there is an option to write with temporaryGcsBucket without the need to pass this config Thank you for your time. Take care!