[SUPPORT] Transform from kafka complains about table not found when using transformer.sql

Dear experts, we are trying to read data from a kafka topic and write it with hudi applying a transformation. Note our hudi command has worked before without transformation. Also we are working on a cluster that is configured very well and overall working fine.

**Command used" the relevant part of the spark-submit command is:

spark-submit
--jars abfs://somelocation/hudi-utilities-bundle_2.11-0.6.0.jar \
--props /somepath/mypropfile.properties \
--target-base-path abfs://somepath/somedb/unmanaged/sometable \
--table-type COPY_ON_WRITE \
--transformer-class org.apache.hudyi.utilities.transform.SqlQueryBasedTransformer \
--hoodie-conf hoodie.deltastreamer.transformer.sql='select `fielda,fieldb from <SRC>';

Stacktrace We keep on getting following error complaining about the table provided in target-base-path:

ERROR yarn.Client: Application diagnostics message: User class threw exception: org.apache.hudi.exception.TableNotFoundException: Hoodie table not found in path abfs://somepath/somedb/unmanaged/sometable/.hoodie'

We have defined an external table “sometable” for the table to land in the location above (abfs://somepath/somedb/unmanaged/sometable/), and it is empty so that hudi can write the data there.

Questions I have 1)in the “hoodie.deltastreamer.transformer.sql” statement, the SELECT … FROM < SRC > -> what table do we choose here? do we really write < SRC > in the code? Does it know then from the props file that you have provided that he needs to fetch it from a kafka topic? 2)is our approach correct: we have a working kafka topic, and an empty external table, with the location that we provide in target-base-path hudi should read the data from the kafka topic and then write to the location of the external table?

Environment Description

• Hudi version : 0.6.0

• Spark version :2.4.5

• Hive version :Hive 3.1 on Tez

• Hadoop version :3.1.1

• Storage (HDFS/S3/GCS…) : abfs (azure blob storage)

• Running on Docker? (yes/no) :no

[xushiyan]

@JB-data I have not used it myself. but from the code and example here, you write <SRC> which will be interpreted. See

• org.apache.hudi.utilities.transform.TestSqlQueryBasedTransformer
• org.apache.hudi.utilities.transform.SqlFileBasedTransformer#SRC_PATTERN

[xushiyan]

This is similar to https://github.com/apache/hudi/issues/3906

@JB-data can you try not manually creating the path before job starts? deltastreamer is supposed to init a hoodie table for you. Also please consider upgrading to 0.8 or later since 0.6 is a bit too old.