Dynamodb batch writer periodically failing

I’m using the following logic to pull content from a list and insert it into DynamoDB

dynamodb = boto3.resource("dynamodb")
keys_table = dynamodb.Table("my-dynamodb-table")

with keys_table.batch_writer() as batch:
        for key in objects[tmp_id]:
            batch.put_item(Item={
                    "cluster": cluster,
                    "tmp_id": tmp_id,
                    "manifest": manifest_key,
                    "key": key,
                    "timestamp": timestamp
            })

It appears to periodically append more than the 25 item limit to the batch and thus fails with the following error:

  File "/Library/Python/2.7/site-packages/boto3/dynamodb/table.py", line 121, in __exit__
    self._flush()
  File "/Library/Python/2.7/site-packages/boto3/dynamodb/table.py", line 102, in _flush
    RequestItems={self._table_name: self._items_buffer})
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 383, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 425, in _convert_to_request_dict
    api_params, operation_model)
  File "/Library/Python/2.7/site-packages/botocore/validate.py", line 273, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid length for parameter RequestItems.my-dynamodb-table, value: 26, valid range: 1-25

Package version:

Python 2.7.10
boto3==1.2.2
botocore==1.3.12

[jamesls]

The ProvisionedThroughputExceededException is being retried (https://github.com/boto/botocore/blob/develop/botocore/data/_retry.json#L78-L94). What might be happening is that all the 10 retry attempts are being exhausted at which point we’ll propagate the ProvisionedThroughputExceededException exception.

I think we may need to allow the user to override the max number of retry attempts so users can set it to a higher number as needed.

[ggtools]

I don’t know if that might be included in the fix but I faced this issue when working a heavily loaded environment. In addition sending more than 25 operations in the batch I also noticed that at some point DynamoDB was refusing the batch with ProvisionedThroughputExceededException.

While this might be the responsibility of the caller during put_item it might be tricky for the call to deal with the errors during the final flush which is performed in __exit__

Does it makes sense to catch the ClientError with the error code ProvisionedThroughputExceededException and consider it as no element has been flushed?