Getting error ResourceNumberLimitExceededException) when calling the UpdateTable operation: Number of TABLE_VERSION resources exceeds the limit 100000 per TABLE

P.S. Don’t attach file. Please, prefer add code snippets directly in the message body. I have the following code: wr.s3.to_parquet(df=df, path=destination_path, dtype=types, dataset=True, database=DATABASE, table=TABLE, mode="append", catalog_versioning=False, partition_cols=['env','year','month','day','hour']) I am executing this on S3 event when data file created in s3. At some point I got error “(ResourceNumberLimitExceededException) when calling the UpdateTable operation: Number of TABLE_VERSION resources exceeds the limit 100000 per TABLE” . Is there any way not to update table version? The table stay the same, only partitions are added with new data.

[janpio]

For anyone else finding this issue years later:

This error message is related to a limit in AWS Glue called Max table versions per table. It doesn’t seem to be documented anywhere, but fortunately is listed on the Service Quotas page and you can request an increase: https://us-east-1.console.aws.amazon.com/servicequotas/home/services/glue/quotas/L-071ABE08 (Make sure to select the correct Region in the top right dropdown!) It seems to be one of those quotas that can easily increased 10x in seconds 👍

[edvorkin]

@igorborgest I am using pip install awswrangler without specifying any version. I see in the path it is awswrangler-1.7.0.dist-info.