NullPointerException in Kafka Connect

Environment

• OS version: Centos 7.5
• JDK version: AdoptOpenJDK 11.0.4
• ClickHouse Server version: 21.5.5.12
• ClickHouse Native JDBC version: 2.5.5-shaded
• (Optional) Spark version: N/A
• (Optional) Other components’ version: kafkaconnect version 6.1.1-2.12

Error logs

org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
	at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:614)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:189)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:238)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NullPointerException
	at com.github.housepower.jdbc.data.type.DataTypeInt64.serializeBinary(DataTypeInt64.java:77)
	at com.github.housepower.jdbc.data.Column.write(Column.java:31)
	at com.github.housepower.jdbc.data.Block.appendRow(Block.java:93)
	at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.addParameters(ClickHousePreparedInsertStatement.java:162)
	at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.addBatch(ClickHousePreparedInsertStatement.java:95)
	at io.confluent.connect.jdbc.sink.PreparedStatementBinder.bindRecord(PreparedStatementBinder.java:115)
	at io.confluent.connect.jdbc.sink.BufferedRecords.flush(BufferedRecords.java:184)
	at io.confluent.connect.jdbc.sink.JdbcDbWriter.write(JdbcDbWriter.java:80)
	at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:84)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:586)
	... 10 more

Steps to reproduce

The message contains types (as per CH types): String, UInt64, Float64, DateTime. Some columns can be NULL, as is defined in CH schema, but looks like there is some NPE happens inside the driver when kafka message field is actually null.

Other descriptions

The same ingest works with official HTTP protocol driver, but would prefer to use this one as it works better in throughput and resource usage.

[akamensky]

Yes, while it is clearer now and I can ingest the data into clickhouse, one thing could be improved on this driver side is to have a better error handling for this case.

The error here is very generic and does not point in the right direction. The point of having error messages at all is to make it clear what happened and possibly point to correct solution.

I suggest to:

• catch this exception
• and check if the exception happened because of null being received for non-nullable column
• if it is then throw another exception which provides much clearer message (i.e. received NULL value for NOT NULL column or something similar, maybe even include column name in the message if possible).

This will allow users to handle those issues in much cleaner way (and not bothering you with these issues regularly as it seems from issues history for this type of issue).

[pan3793]

Yes, ClickHouse use String to represent VARCHAR NOT NULL in most ANSI compatible database, and Nullable(String) to represent VARCHAR NULL. BTW, it’s by design that throw exception when insert null to a not null column, I will close this issue, and feel free to reopen or open new issues if you encounter other questions.