[Feature][plugin-spark-source-jdbc] add jdbc connector settings in source config for easy used
See original GitHub issueSearch before asking
- I had searched in the feature and found no similar feature requirement.
Description
When I config plugin-spark-source-jdbc for one jdbc source,I found i can’t set jdbc properties through its config file.I looked through the code and found that jdbc source’s config has no entrance to set properties for jdbc connector.The only way you can do this is through the URL like this eg:
jdbc:mysql://localhost:3306/test?connectTimeout=10000&socketTimeout=10000
This url set connectTimeout and socketTimeout for the connector.
Then, I looked through spark’s doc, I found this topic: https://spark.apache.org/docs/2.4.0/sql-data-sources-jdbc.html , in fact that spark supports accepting the parameters of connector. It looks like this:
val connectionProperties = new Properties() connectionProperties.put("user", "username") connectionProperties.put("password", "password") val jdbcDF2 = spark.read .jdbc("jdbc:postgresql:dbserver", "schema.tablename", connectionProperties) // Specifying the custom data types of the read schema connectionProperties.put("customSchema", "id DECIMAL(38, 0), name STRING") val jdbcDF3 = spark.read .jdbc("jdbc:postgresql:dbserver", "schema.tablename", connectionProperties)
Usage Scenario
Add a prefix for connector configtions, jdbc input config like this: jdbc { driver = “com.mysql.jdbc.Driver” url = “jdbc:mysql://localhost:3306/info” table = “access” result_table_name = “access_log” user = “username” password = “password” jdbc.partitionColumn = “item_id” jdbc.numPartitions = “10” jdbc.lowerBound = 0 jdbc.upperBound = 100 conn.connectTimeout = 10000 conn.socketTimeout = 10000 }
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project’s Code of Conduct
Issue Analytics
- State:
- Created 2 years ago
- Reactions:1
- Comments:8 (7 by maintainers)
Top GitHub Comments
I checked the jdbc configuration in spark project. These two
connectTimeout
andsocketTimeout
are belong tourl
configuration item, we can not set them in other places.SeaTunnel
project already supports prefix, the prefix isjdbc
cloused