Spark : The iceberg-spark3-extensions module CI test failed

I found that ,the iceberg-spark3-extensions module CI test often failed.

the exception like this

org.apache.iceberg.spark.extensions.TestCopyOnWriteDelete > testDeleteNonExistingRecords[catalogName = spark_catalog, implementation = org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive, default-namespace=default, clients=1, parquet-enabled=false, cache-enabled=false}, format = avro, vectorized = false] FAILED
    java.lang.RuntimeException: Failed to get table info from metastore default.table

        Caused by:
        org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed)

            Caused by:
            java.net.SocketException: Broken pipe (Write failed)

[stevenzwu]

I started a thread titled as “test flakiness with SocketException of broken pipe in HiveMetaStoreClient” [1] in the dev mailing list two weeks ago. People have offered some insights there. Main suspicious is connection leak problem.

[1] https://www.mail-archive.com/[e-mail]/msg01789.html

[kbendick]

I started a thread titled as “test flakiness with SocketException of broken pipe in HiveMetaStoreClient” [1] in the dev mailing list two weeks ago. People have offered some insights there. Main suspicious is connection leak problem.

[1] https://www.mail-archive.com/[e-mail]/msg01789.html

Cool thank you @stevenzwu. Connection leak would potentially explain why I’m much less able to reproduce when I increase the catalog’s clients config value. Also, knowing that it is (was?) happening in the flink tests helps a ton with guidance for how much time to put into investigating from the POV of a potential test inheritance pattern issue vs connection leak issue.

This is very helpful. I should be more active on the dev list, though I’m only able to contribute somewhat sporadically given my current work is not exactly iceberg related. I’ll follow up in that thread too with anything I find while investigating. 👍