Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

INCREMETNAL QUERY-Null value Exception

See original GitHub issue

Tips before filing an issue

Have you gone through our FAQs? Yes
Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

Facing null value exception while pulling the data from hudi basepath programmatically.

To Reproduce

Steps to reproduce the behavior:

Expected behavior

Get all the records which are being ingested from last begintime to current time incrementally.

Environment Description

Hudi version : 0.5.2
Spark version : 2.2.1
Hive version :
Hadoop version : 2.7
Storage (HDFS/S3/GCS…) : HDFS
Running on Docker? (yes/no) : no

Additional context

Please find the steps and error description. scala> val basepath= “/datalake/xxx/xxx/prd/xxx/hudi_cow_dedup_debug”

scala> spark.read.format(“org.apache.hudi”).load(basepath + “/*”).createOrReplaceTempView(“hudi_tab”)

scala> val commits = spark.sql(“select distinct(_hoodie_commit_time) as commitTime from hudi_tab order by commitTime”).map(k => k.getString(0)).take(50) commits: Array[String] = Array(20200622140846, 20200622151300, 20200622161353, 20200622170749, 20200622180730, 20200622191126, 20200622201001, 20200622211525, 20200622220714, 20200622231112, 20200623001304, 20200623011203, 20200623020759, 20200623031251, 20200623041044)

scala> val beginTime = commits(commits.length - 2) beginTime: String = 20200623031251

scala> val incrementalDF = spark.read.format(“org.apache.hudi”).option(QUERY_TYPE_OPT_KEY, QUERY_TYPE_INCREMENTAL_OPT_VAL).option(BEGIN_INSTANTTIME_OPT_KEY, beginTime).load(basepath); java.lang.NoSuchFieldError: NULL_VALUE at org.apache.hudi.avro.HoodieAvroUtils.addMetadataFields(HoodieAvroUtils.java:143) at org.apache.hudi.avro.HoodieAvroUtils.createHoodieWriteSchema(HoodieAvroUtils.java:130) at org.apache.hudi.IncrementalRelation.<init>(IncrementalRelation.scala:77) at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:80) at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:47) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:307) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:156) … 60 elided

scala> val incrementalDF = spark.read.format(“org.apache.hudi”).option(“hoodie.datasource.query.type”,“incremental”).option(“hoodie.datasource.read.begin.instanttime”,beginTime).load(basepath); java.lang.NoSuchFieldError: NULL_VALUE at org.apache.hudi.avro.HoodieAvroUtils.addMetadataFields(HoodieAvroUtils.java:143) at org.apache.hudi.avro.HoodieAvroUtils.createHoodieWriteSchema(HoodieAvroUtils.java:130) at org.apache.hudi.IncrementalRelation.<init>(IncrementalRelation.scala:77) at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:80) at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:47) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:307) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:156) … 60 elided

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.