Dynamodb batch writer periodically failing
See original GitHub issueI’m using the following logic to pull content from a list and insert it into DynamoDB
dynamodb = boto3.resource("dynamodb")
keys_table = dynamodb.Table("my-dynamodb-table")
with keys_table.batch_writer() as batch:
for key in objects[tmp_id]:
batch.put_item(Item={
"cluster": cluster,
"tmp_id": tmp_id,
"manifest": manifest_key,
"key": key,
"timestamp": timestamp
})
It appears to periodically append more than the 25 item limit to the batch and thus fails with the following error:
File "/Library/Python/2.7/site-packages/boto3/dynamodb/table.py", line 121, in __exit__
self._flush()
File "/Library/Python/2.7/site-packages/boto3/dynamodb/table.py", line 102, in _flush
RequestItems={self._table_name: self._items_buffer})
File "/Library/Python/2.7/site-packages/botocore/client.py", line 310, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Library/Python/2.7/site-packages/botocore/client.py", line 383, in _make_api_call
api_params, operation_model, context=request_context)
File "/Library/Python/2.7/site-packages/botocore/client.py", line 425, in _convert_to_request_dict
api_params, operation_model)
File "/Library/Python/2.7/site-packages/botocore/validate.py", line 273, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid length for parameter RequestItems.my-dynamodb-table, value: 26, valid range: 1-25
Package version:
Python 2.7.10
boto3==1.2.2
botocore==1.3.12
Issue Analytics
- State:
- Created 8 years ago
- Reactions:1
- Comments:5 (3 by maintainers)
Top Results From Across the Web
Error handling with DynamoDB - AWS Documentation
The most likely cause of a failed read or a failed write is throttling. For BatchGetItem , one or more of the tables...
Read more >Lambda to Dynamo Write: Intermittently missing batch data
Hi, I am using AWS Lamba to read data from web and store it in dynamoDB at periodic intervals ... Lambda to Dynamo...
Read more >Error at trying to write batch item in dynamodb with Mapvalues
The field2 can be formed like this. The DynamoDB will automatically interpret it as MAP (i.e. no need to specifically mention 'M').
Read more >7 Mistakes I Made In DynamoDB - Coder Diaries
Scanning the data is expensive. Consider using query over scan to get the data. Secondary Index is considered another ARN resource. Batch Write...
Read more >Getting response of AWS DynamoDB BatchWriter request
By the way, you should be aware of the success and failure of batch_write_item request. Partial success or failure of the items results...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
The
ProvisionedThroughputExceededException
is being retried (https://github.com/boto/botocore/blob/develop/botocore/data/_retry.json#L78-L94). What might be happening is that all the 10 retry attempts are being exhausted at which point we’ll propagate theProvisionedThroughputExceededException
exception.I think we may need to allow the user to override the max number of retry attempts so users can set it to a higher number as needed.
I don’t know if that might be included in the fix but I faced this issue when working a heavily loaded environment. In addition sending more than 25 operations in the batch I also noticed that at some point DynamoDB was refusing the batch with
ProvisionedThroughputExceededException
.While this might be the responsibility of the caller during
put_item
it might be tricky for the call to deal with the errors during the final flush which is performed in__exit__
Does it makes sense to catch the
ClientError
with the error codeProvisionedThroughputExceededException
and consider it as no element has been flushed?