Persist raw data from kafka topic as it is

Feature request: To save raw data from topic in pinot table.

Use case : We have lots of complex schemas and we are using pinot for saving and retreiving topic data with times stamp and some other fields. We do not want to map all nested columns from complex schema and create pinot schema and use lots of transformation functions. There are some places we want raw data as it is in pinot table.

Sample data :

{ "header": { "tid": "12wee", "rid": 1, "timestamp": 1647347092337 }, "status": "200_SUCCESS", "jasData": { "sdata": -22.89122, "cnn": 0.823469, "kli": 2.238848, "olp": [ { "ovPerc": 0.032486767, "hg": 30.0, "abshi": 6.661863 } ], "terrkl": { "ovPerc": 0.9675132, "dist": [ -25.17232, -25.17232, -25.130081 ] }, "bcut": 2.77 }, "rgData": { "pre": 102033.33, "pv": 0.16, "t": 287.36, "timestamp": 1647347069000 }, "timestamp": 1647347092337 } }

[saumya2700]

@Jackie-Jiang This sounds good to me. @saumya2700 If you’re not working on this, I can pick this up?

yes Please go ahead.

[Jackie-Jiang]

We may consider adding a new config in the IngestionConfig to store the json string of the record into a field. The logic needs to be implemented into the RecordExtractor