RecursionError using batch writer

For some reason, this works fine with a regular put_item call, but not if I pass the same item to the batch writer:

#!/usr/bin/env python3

import decimal
import os
import pprint
import random

import boto3
from faker import Faker
from faker.providers import company

dynamodb_client = boto3.client('dynamodb')
dynamodb = boto3.resource('dynamodb')
fake = Faker()
pp = pprint.PrettyPrinter(indent=2)

table_name = 'my_table'
existing_tables = dynamodb_client.list_tables()['TableNames']

# Create the DynamoDB table if it does not exist

if table_name not in existing_tables:
    response = dynamodb_client.create_table(
        TableName=table_name,
        KeySchema=[
            {
                'AttributeName': 'artist_id',
                'KeyType': 'HASH',
            },
            {
                'AttributeName': 'album_id',
                'KeyType': 'RANGE',
            },
        ],
        AttributeDefinitions=[
            {
                'AttributeName': 'artist_id',
                'AttributeType': 'N',
            },
            {
                'AttributeName': 'album_id',
                'AttributeType': 'N',
            },
        ],
        ProvisionedThroughput={
            'ReadCapacityUnits': 50,
            'WriteCapacityUnits': 500,
        }
    )

    print('Waiting for table ', table_name, 'to create...')
    waiter = dynamodb_client.get_waiter('table_exists')
    waiter.wait(TableName=table_name)

ddb_table = dynamodb.Table(table_name)

album_id = 0

album_formats = ['7" vinyl', '10" Vinyl', '12" Vinyl']

# Create 1000 artists, each with 10 albums

with ddb_table.batch_writer() as batch:

    for artist_id in range(1, 1001):

        item = {}
        item['artist_id'] = artist_id
        item['artist_name'] = fake.name()

        for album in range(10):
            album_id = album_id + 1
            item['album_id'] = album_id
            item['album_title'] = fake.catch_phrase().title()
            item['album_meta'] = {}
            item['album_meta']['year'] = fake.year()
            item['album_meta']['sku'] = f'{artist_id}-{album_id}'
            item['album_meta']['format'] = random.choice(album_formats)
            item['album_meta']['price'] = decimal.Decimal(str(
                round(random.uniform(3.00, 50.00), 2)))

            id_album = list(str(album_id))
            id_album.reverse()
            filepath = '/'.join(id_album)
            filename = os.path.join(filepath, f'{album_id}.jpg')
            s3_uri = f'/albumart/{filepath}/{album_id}.jpg'

            item['track_names'] = []
            item['album_meta']['s3_uri'] = s3_uri
            item['album_meta']['tracks'] = []

            # each album has 12 tracks

            for track_id in range(1, 13):
                item['track_names'].append(f'Track{track_id}')

                track = {}
                track['name'] = f'Track{track_id}'
                track['length'] = round(random.uniform(30000, 300000))
                track['name'] = f'Track{track_id}'
                track['position'] = f'{track_id}'
                item['album_meta']['tracks'].append(track)

            print(album_id)
            pp.pprint(item)
            batch.put_item(Item=item)

Here is the abbreviated stack trace:

  File "/usr/local/lib/python3.7/site-packages/boto3/dynamodb/types.py", line 231, in _serialize_m
    return dict([(k, self.serialize(v)) for k, v in value.items()])
  File "/usr/local/lib/python3.7/site-packages/boto3/dynamodb/types.py", line 231, in <listcomp>
    return dict([(k, self.serialize(v)) for k, v in value.items()])
  File "/usr/local/lib/python3.7/site-packages/boto3/dynamodb/types.py", line 102, in serialize
    dynamodb_type = self._get_dynamodb_type(value)
  File "/usr/local/lib/python3.7/site-packages/boto3/dynamodb/types.py", line 124, in _get_dynamodb_type
    elif self._is_type_set(value, self._is_number):
  File "/usr/local/lib/python3.7/site-packages/boto3/dynamodb/types.py", line 183, in _is_type_set
    if self._is_set(value):
  File "/usr/local/lib/python3.7/site-packages/boto3/dynamodb/types.py", line 178, in _is_set
    if isinstance(value, collections_abc.Set):
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/abc.py", line 139, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison

Using boto3 1.9.205 on macOS 10.14.6 with Python 3.7.4.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don’t have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[swetashre]

@mrichman - Yes it will defeat the purpose of batching altogether. That was only a workaround. The problem in your case is that the same item object is getting reused each time when you call put_item. What’s happening is that you have a list of items you want to batch but they’re all referencing the same object, so what happens is we iterate through each item trying to transform it, but because they’re all the same reference, they get transformed over and over again for each element in the list

The fix for your use case is to just create a new item dictionary each time:

with ddb_table.batch_writer() as batch:

    for artist_id in range(1, 1001):


        for album in range(10):
            item = {}
            item['artist_id'] = artist_id
            item['artist_name'] = 'hi'
            album_id = album_id + 1
            item['album_id'] = album_id
            ...

Can you please try this solution and let me know if it helps or not ?