Hive read issues when different partition have different schemas.

Hive reads writer schema separately for each partition. If a schema has evolved and updates has not made for all partitions (i.e. for some partition last change was from older schema), they hive read for that partition would fail since non availability of new column in schema.

Concerned code: (Class: AbstractRealtimeRecordReader)

private void init() throws IOException {
    Schema schemaFromLogFile = LogReaderUtils.readLatestSchemaFromLogFiles(split.getBasePath(), split.getDeltaLogPaths(), jobConf);
    if (schemaFromLogFile == null) {
      writerSchema = InputSplitUtils.getBaseFileSchema((FileSplit)split, jobConf);
      LOG.info("Writer Schema From Parquet => " + writerSchema.getFields());
    } else {
      writerSchema = schemaFromLogFile;
      LOG.info("Writer Schema From Log => " + writerSchema.toString(true));
    }

I tried replacing this writer schema to get schema from TableSchemaResolver. This is working fine for me.

Does this look good? I haven’t followed hive flow in detail yet.

[aditiwari01]

Have quite a busy sprint. I’ll try to get back. If not, I’ll update by weekend.

[nsivabalan]

able to get hold of it, thanks. will close the github issue and follow up on the patch review. thanks!