Add ConditionExpression to dynamodb batch writer

I have a use case where I want to write a few dozen rows to dynamodb at a time, with conditions.

Use Case

But there’s a certain edge case I’m trying to handle, where I’m trying to write two sets of data to the table which describe the same thing, but one is more recent (and therefore more accurate) than the other. So I need to use dynamodb conditions to say:

I can do this with a for loop over

response = table.put_item(
            Item=row,
            ReturnValues='ALL_OLD',
            ReturnConsumedCapacity='NONE',
            ConditionExpression=cond
        )

Including the server-side atomic logic, this is basically:

for new_row in rows:
   if new_row not already in table:
      put_item(new_row) # new
   elif existing_row['t'] < new_row['t']:
      put_item() # overwite because the new row is more recent
   else:
      pass
      # do nothing, because the data in the table is newer than what we're trying to write

This is pretty slow, because there’s a separate HTTP call for each row.

I would like to use the batch writer for this, but it currently doesn’t accept Conditions.

MWE

import boto3
from boto3.dynamodb.conditions import Key, Attr

table_name = 'test_batch_write'
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table(table_name)

for delta in [0, 1]:
    with table.batch_writer() as batch:
        for i in range(10):
            val = i + delta * (i%2)
            cond = Attr('regionid').not_exists() & Attr('interval_end').not_exists() # row doesn't yet exist
            cond |= Attr('val').lt(val) # or val is lower
            print(f"Put {i}")
            batch.put_item(
                Item={
                    'top_timestamp': i,
                    'regionid': 'QLD1',
                    'val': val
                },
                ConditionExpression=cond)

Actual Result:

Traceback (most recent call last):
  File "batch_write.py", line 21, in <module>
    ConditionExpression=cond)
TypeError: put_item() got an unexpected keyword argument 'ConditionExpression'

Expected result:

Dynamodb contains 10 rows. For each row with an even i, the val column equals i which equals top_timestamp. For each other row, the val column equals i+1.

Note that with individual put calls, if the condition fails a boto3.client('dynamodb').exceptions.ConditionalCheckFailedException exception is throw which has to be caught inside each iteration over my for loop. I’m not sure what the best way to handle that for a batch write is.

[rrhodes]

+1 for this feature, it would be hugely helpful in my work.

Took a quick look into this. It appears we need to add support for ConditionExpression into the batch_write_item method before it’s possible to fully satisfy this request. Said method doesn’t appear to be open-source, or at least I can’t find it, otherwise I would have been happy to consider adding this feature myself.

[dazzag24]

+1 for this idea. I have to update a table with incremental updates and this would allow me to avoid a whole bunch of client side logic to decide if a item already exists or not.