Error ArrayIndexOutOfBoundsException while inserting array

Environment

• OS version: Linux CentOS 7
• JDK version: 8
• ClickHouse Server version: N/A (latest)
• ClickHouse Native JDBC version: 2.5.4
• (Optional) Spark version: 2.4.3 (cluster cdh-6)
• (Optional) Other components’ version: N/A

Error logs

java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 1048576
	at com.github.housepower.repackaged.net.jpountz.util.SafeUtils.checkRange(SafeUtils.java:24)
	at com.github.housepower.repackaged.net.jpountz.util.SafeUtils.checkRange(SafeUtils.java:32)
	at com.github.housepower.repackaged.net.jpountz.lz4.LZ4JavaSafeCompressor.compress(LZ4JavaSafeCompressor.java:141)
	at com.github.housepower.repackaged.net.jpountz.lz4.LZ4Compressor.compress(LZ4Compressor.java:95)
	at com.github.housepower.jdbc.buffer.CompressedBuffedWriter.flushToTarget(CompressedBuffedWriter.java:75)
	at com.github.housepower.jdbc.serde.BinarySerializer.flushToTarget(BinarySerializer.java:112)
	at com.github.housepower.jdbc.connect.NativeClient.disconnect(NativeClient.java:153)
	at com.github.housepower.jdbc.ClickHouseConnection.close(ClickHouseConnection.java:141)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:710)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:834)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:834)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2121)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2121)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Steps to reproduce

Insert Array of String into Clickhouse table using Yarn cluster.

Other descriptions

I’ve seen that that error “got” fixed, but as you can see it’s not the case : I’m using shaded versions of plugin too.

Created simple UDF to parse string into Array and then insert it -

  val toArr = udf((value: String) =>
    Option(value) match {
      case None  => Array[String]()
      case Some(value) =>
        if (value.contains(",")) {
          val res = value.split(",", -1).map(_.trim)
          res
        }
        else Array(value)
    })

But i get this error eventually.

[ccellado]

Ok, I’ve worked around it.

The problem was not the data per se, but using arrays in big scoped migrations.

1. Cloned master with the latest lz4 library changed to Aircompressor @pan3793
2. Built it with spark-integration for scala 2.11 and spark 2.4.3 (had to change pom files and remove spark-3 version)
3. Got new problem broken pipe error. Changed batchsize from 500_000 to 10_000 and got it working!

The big question: was it only the change in batch size or does the lz4 library helped too? Maybe i’ll have time to test it later.

Thanks for the cool plugin!

[ccellado]

Well, in my experience 500000 does not work unfortunately with Arrays.

Forgot to mention - when there were no complex types like Array, 500000 batch went fine.