batch insert meet nullpoint exception

Environment

• OS version: mac lastest
• JDK version: jdk 11
• ClickHouse Server version: lastest
• ClickHouse Native JDBC version: lastest
• (Optional) Spark version: N/A 3.0.2
• (Optional) Other components’ version: N/A

I think maybe the spark dataframe rows size is zero

Error logs

Caused by: java.lang.NullPointerException
        at com.github.housepower.jdbc.data.type.DataTypeInt64.serializeBinary(DataTypeInt64.java:77)
        at com.github.housepower.jdbc.data.Column.write(Column.java:31)
        at com.github.housepower.jdbc.data.Block.appendRow(Block.java:93)
        at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.addParameters(ClickHousePreparedInsertStatement.java:162)
        at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.addBatch(ClickHousePreparedInsertStatement.java:95)
        at com.hw.utils.CKJDBCUtils$.saveBatchRowInsertDfToCK(CKJDBCUtils.scala:724)
        at com.hw.utils.CKJDBCUtils$.insertIterRowToCKTable(CKJDBCUtils.scala:691)
        at com.hw.utils.CKJDBCUtils$.$anonfun$initTableWithInsertDataByRDDRows$1(CKJDBCUtils.scala:428)
        at com.hw.utils.CKJDBCUtils$.$anonfun$initTableWithInsertDataByRDDRows$1$adapted(CKJDBCUtils.scala:425)
        at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:994)
        at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:994)
        at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2154)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:127)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:462)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:465)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

Steps to reproduce

Other descriptions

[TimmannaC]

Hi Pan, Thank you for you inputs. I was able to resolve the issue. I was using ReplacingMergeTree() Engine which was not allowing to write duplicate records.

thank you for your support.

[lala7573]

@TimmannaC Could you give me any advice about registerDialect in pyspark? In JdbcDialects.registerDialect(ClickHouseDialect), the ClickHouseDialect is an object (not a class) so I can’t find out how to do it.

The below code doesn’t work 😢 … it said 'JavaPackage' object is not callable.

from py4j.java_gateway import java_import

java_import(jvm, "org.apache.spark.sql.jdbc.ClickHouseDialect")
gw.jvm.org.apache.spark.sql.jdbc.JdbcDialects.registerDialect(
    gw.jvm.org.apache.spark.sql.jdbc.ClickHouseDialect
)

And I found out how to get reference of scala object, but it also doesn’t work, and the error message said ClassNotFoundException: org.apache.spark.sql.jdbc.ClickHouseDialect$.

def ref_scala_object(jvm, object_name):
    clazz = jvm.java.lang.Class.forName(object_name+"$")
    ff = clazz.getDeclaredField("MODULE$")
    o = ff.get(None)
    return o