When using hiveCatalog.dropTable(identifier, true), the table directory is not completely removed

When using hiveCatalog.dropTable(identifier, true) to drop a Iceberg table, the table directory is not completely removed. eg. before deleting the table, the data directory of the table is as follows:

⇒  tree   /data/hive/warehouse/test/
/data/hive/warehouse/test/
├── data
│   └── ts_year=2020
│       ├── id_bucket=0
│       │   ├── 00000-0-4718ae1d-ee92-4a39-9c00-6225e791cc68-00001.parquet
│       │   ├── 00000-0-88059c29-5b0d-44da-a7e2-fd886f6ff04a-00001.parquet
│       │   ├── 00001-1-5ea45c12-b7e4-47f3-8fc3-c12849679b9f-00002.parquet
│       │   └── 00001-1-6c8baddb-d0dc-4c49-9d89-75e3c55bac83-00002.parquet
│       └── id_bucket=1
│           ├── 00001-1-5ea45c12-b7e4-47f3-8fc3-c12849679b9f-00001.parquet
│           └── 00001-1-6c8baddb-d0dc-4c49-9d89-75e3c55bac83-00001.parquet
└── metadata
    ├── 00000-aaa7a3d5-bb25-4d07-b28c-1e9b63ef8380.metadata.json
    ├── 00001-77ec6836-5709-44d6-a8aa-405588cc93df.metadata.json
    ├── 00002-21692627-9ba1-47ef-8729-d9cd96533ba5.metadata.json
    ├── 57b07bcc-e3a1-4684-a2b3-26263f2b0535-m0.avro
    ├── bff1d409-7d63-4b33-9d10-d7ebe7efe65c-m0.avro
    ├── snap-2855480055257649189-1-57b07bcc-e3a1-4684-a2b3-26263f2b0535.avro
    └── snap-8307457369176907400-1-bff1d409-7d63-4b33-9d10-d7ebe7efe65c.avro

5 directories, 13 files

after drop table, the data directory of the table is as follows:

⇒  tree   /data/hive/warehouse/test/
/data/hive/warehouse/test/
├── data
│   └── ts_year=2020
│       ├── id_bucket=0
│       └── id_bucket=1
└── metadata
    ├── 00000-aaa7a3d5-bb25-4d07-b28c-1e9b63ef8380.metadata.json
    └── 00001-77ec6836-5709-44d6-a8aa-405588cc93df.metadata.json

5 directories, 2 files

I think the other two meta files should also be deleted, because these files are actually useless, the ts_year=2020/id_bucket=0 and ts_year=2020/id_bucket=1 directories should also need to be deleted.

If drop iceberg table by hadoop catalog or spark.sql("drop table xxx"), all directories associated with the table will be deleted. we should make these behaviors consistent.

[rdblue]

@zhangdove, there is no requirement in Iceberg that a table “owns” its location. The intent of not recursively deleting was to avoid dropping other data in the same prefix. I think it would be fine to drop the location recursively in some cases. Maybe that should be a catalog option?

[397090770]

Thanks for your reply, I will submit an RP to make these behaviors consistent.