question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Feature][plugin-spark-source-jdbc] add jdbc connector settings in source config for easy used

See original GitHub issue

Search before asking

  • I had searched in the feature and found no similar feature requirement.

Description

When I config plugin-spark-source-jdbc for one jdbc source,I found i can’t set jdbc properties through its config file.I looked through the code and found that jdbc source’s config has no entrance to set properties for jdbc connector.The only way you can do this is through the URL like this eg: jdbc:mysql://localhost:3306/test?connectTimeout=10000&socketTimeout=10000 This url set connectTimeout and socketTimeout for the connector. Then, I looked through spark’s doc, I found this topic: https://spark.apache.org/docs/2.4.0/sql-data-sources-jdbc.html , in fact that spark supports accepting the parameters of connector. It looks like this: val connectionProperties = new Properties() connectionProperties.put("user", "username") connectionProperties.put("password", "password") val jdbcDF2 = spark.read .jdbc("jdbc:postgresql:dbserver", "schema.tablename", connectionProperties) // Specifying the custom data types of the read schema connectionProperties.put("customSchema", "id DECIMAL(38, 0), name STRING") val jdbcDF3 = spark.read .jdbc("jdbc:postgresql:dbserver", "schema.tablename", connectionProperties)

Usage Scenario

Add a prefix for connector configtions, jdbc input config like this: jdbc { driver = “com.mysql.jdbc.Driver” url = “jdbc:mysql://localhost:3306/info” table = “access” result_table_name = “access_log” user = “username” password = “password” jdbc.partitionColumn = “item_id” jdbc.numPartitions = “10” jdbc.lowerBound = 0 jdbc.upperBound = 100 conn.connectTimeout = 10000 conn.socketTimeout = 10000 }

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:1
  • Comments:8 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
leo65535commented, Dec 21, 2021

I checked the jdbc configuration in spark project. These two connectTimeout and socketTimeout are belong to url configuration item, we can not set them in other places.

Add a prefix for connector configtions

SeaTunnel project already supports prefix, the prefix is jdbc

image

0reactions
CalvinKirscommented, Mar 25, 2022

cloused

Read more comments on GitHub >

github_iconTop Results From Across the Web

JDBC Source Connector Configuration Properties
Comma-separated list of one or more timestamp columns to detect new or modified rows using the COALESCE SQL function. Rows whose first non-null...
Read more >
JDBC To Other Databases - Spark 3.3.1 Documentation
Property Name Default Scope url (none) read/write dbtable (none) read/write query (none) read/write
Read more >
Configuring a JDBC provider and data source - IBM
You must set the appropriate properties on every data source that you configure. Configuring a JDBC provider using the administrative console
Read more >
Setting Up a JDBC Connection to the Data Source
Setting Up a JDBC Connection to the Data Source · From the Administration page, click JDBC Connection to display the list of existing...
Read more >
Configure the Databricks ODBC and JDBC drivers
ODBC configuration and connection parameters. Install and configure the ODBC driver for Windows. In Windows, you can set up a Data Source Name...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found