HoodieCommitArchiveLog.openWriter causes object store to throw exceptions

Ran into following issue when trying to commit a batch into google cloud storage and it always fails for the first time and somehow powered through on the retry

com.uber.hoodie.exception.HoodieException: Unable to initialize HoodieLogFormat writer
    at com.uber.hoodie.io.HoodieCommitArchiveLog.openWriter(HoodieCommitArchiveLog.java:93)
    at com.uber.hoodie.io.HoodieCommitArchiveLog.archiveIfRequired(HoodieCommitArchiveLog.java:115)
    at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:541)
    at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:488)
    at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:479)
    at com.uber.hoodie.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:162)
    at com.uber.hoodie.HoodieStreamingSink$$anonfun$1$$anonfun$apply$2.apply(HoodieStreamingSink.scala:51)
    at com.uber.hoodie.HoodieStreamingSink$$anonfun$1$$anonfun$apply$2.apply(HoodieStreamingSink.scala:51)
    at scala.util.Try$.apply(Try.scala:192)
    at com.uber.hoodie.HoodieStreamingSink$$anonfun$1.apply(HoodieStreamingSink.scala:50)
    at com.uber.hoodie.HoodieStreamingSink$$anonfun$1.apply(HoodieStreamingSink.scala:56)
    at com.uber.hoodie.HoodieStreamingSink.retry(HoodieStreamingSink.scala:99)
    at com.uber.hoodie.HoodieStreamingSink.addBatch(HoodieStreamingSink.scala:49)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$3$$anonfun$apply$16.apply(MicroBatchExecution.scala:475)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$3.apply(MicroBatchExecution.scala:473)
    at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:271)
    at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch(MicroBatchExecution.scala:472)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:133)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:121)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:121)
    at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:271)
    at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1.apply$mcZ$sp(MicroBatchExecution.scala:121)
    at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:117)
    at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:279)
    at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)
Caused by: java.io.IOException: The append operation is not supported.
    at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.append(GoogleHadoopFileSystemBase.java:811)
    at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1176)
    at com.uber.hoodie.common.table.log.HoodieLogFormatWriter.<init>(HoodieLogFormatWriter.java:73)
    at com.uber.hoodie.common.table.log.HoodieLogFormat$WriterBuilder.build(HoodieLogFormat.java:204)
    at com.uber.hoodie.io.HoodieCommitArchiveLog.openWriter(HoodieCommitArchiveLog.java:88)
    ... 28 more

[vinothchandar]

Right fix may be just to special case s3a and gs, instead of messing with exception messages… Let me prep a diff based on this and add you

[leletan]

• All micro batches first fail and retry succeeds (or)

what I observed is this one. Did not drill down too much why we are seeing that behavior though.