Add Spark JDBC read

For enterprise use, I’d like to poll the extension of read methods to JDBC, given that drivers are available in the Spark Context.

Current Solutions

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

import databricks.koalas as ks

jdbc_options = dict()
jdbc_options["driver"] = "<my-driver>"

# JDBC
df = ks.DataFrame(
    spark.read.format("jdbc").options(**jdbc_options).option("dbtable", "<sql>").load()
)

# Snowflake
sf_options = dict()
df = ks.DataFrame(
    spark.read.format("net.snowflake.spark.snowflake")
    .options(**sf_options)
    .option("dbtable", "<sql>")
    .load()
)

New Solution

import databricks.koalas as ks

df = ks.read_jdbc(dbtable="<sql>", driver="<my-driver>", **options)

[vaibhavsingh007]

Yes, that seems to work. I was able to create ks dataframe using OP’s method above:

df = ks.DataFrame(
    spark.read.format("net.snowflake.spark.snowflake")
    .options(**sf_options)
    .option("dbtable", "<sql>")
    .load()
)

[vaibhavsingh007]

Thanks @HyukjinKwon I did check that doc before though it doesn’t say how to pass the driver. Should I just create a separate Spark session with driver config (spark.jars) and Koalas will detect that Spark session automatically instead of creating its own?