Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

dynamodb: "IndexError: list index out of range" with hash_key_expression

See original GitHub issue

I’m seeing an IndexError with dynamodb2.responses.query:

key_condition_expression: {'format': '({0} {operator} {1})', 'operator': 'AND', 'values': (<boto3.dynamodb.conditions.Equals object at 0x7f698d034a20>, <boto3.dynamodb.conditions.Between object at 0x7f698d027d30>)}
QUERY: {'ExpressionAttributeNames': {'#date': 'date'},
 'KeyConditionExpression': {'format': '({0} {operator} {1})',
                            'operator': 'AND',
                            'values': ({'format': '{0} {operator} {1}',
                                        'operator': '=',
                                        'values': ('Key: organization',
                                                   'XXX')},
                                       {'format': '{0} {operator} {1} AND {2}',
                                        'operator': 'BETWEEN',
                                        'values': ('Key: date',
                                                   20160123,
                                                   20160124)})},
 'ProjectionExpression': '#date,foobar'}
Calling dynamodb:query with {'KeyConditionExpression': <boto3.dynamodb.conditions.And object at 0x7f698d034518>, 'TableName': 'foobar-baz-test', 'ProjectionExpression': '#date,foobar', 'ExpressionAttributeNames': {'#date': 'date'}}
Resetting dropped connection: dynamodb.XXX.amazonaws.com
Internal Server Error: /api/metrics/foobar
Traceback (most recent call last):
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "…/pyenv/project/lib/python3.5/site-packages/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "…/pyenv/project/lib/python3.5/site-packages/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib64/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "…/pyenv/project/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/rest_framework/views.py", line 466, in dispatch
    response = self.handle_exception(exc)
  File "…/pyenv/project/lib/python3.5/site-packages/rest_framework/views.py", line 463, in dispatch
    response = handler(request, *args, **kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/rest_framework/decorators.py", line 53, in handler
    return func(*args, **kwargs)
  File "…/app/views.py", line 1455, in metrics
    org, start, end)
  File "…/app/metrics.py", line 134, in get_metrics_from_dynamodb
    response = table.query(**query_kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/boto3/resources/factory.py", line 455, in do_action
    response = action(self, *args, **kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/boto3/resources/action.py", line 79, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/client.py", line 396, in _make_api_call
    operation_model, request_dict)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/endpoint.py", line 111, in make_request
    return self._send_request(request_dict, operation_model)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/endpoint.py", line 140, in _send_request
    success_response, exception):
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/endpoint.py", line 213, in _needs_retry
    caught_exception=caught_exception)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/hooks.py", line 226, in emit
    return self._emit(event_name, kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/hooks.py", line 209, in _emit
    response = handler(**kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/retryhandler.py", line 250, in __call__
    caught_exception)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/retryhandler.py", line 265, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/retryhandler.py", line 313, in __call__
    caught_exception)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(attempt_number, caught_exception)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/retryhandler.py", line 355, in _check_caught_exception
    raise caught_exception
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/endpoint.py", line 168, in _get_response
    proxies=self.proxies, timeout=self.timeout)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py", line 370, in send
    timeout=timeout
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "…/pyenv/project/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 374, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.5/http/client.py", line 1170, in getresponse
    response = self.response_class(self.sock, method=self._method)
  File "/usr/lib64/python3.5/http/client.py", line 220, in __init__
    self.fp = sock.makefile("rb")
  File "…/pyenv/project/lib/python3.5/site-packages/httpretty/core.py", line 332, in makefile
    self._entry.fill_filekind(self.fd)
  File "…/pyenv/project/lib/python3.5/site-packages/httpretty/core.py", line 591, in fill_filekind
    status, headers, self.body = self.callable_body(self.request, self.info.full_url(), headers)
  File "…/moto/moto/core/responses.py", line 95, in dispatch
    return cls()._dispatch(*args, **kwargs)
  File "…/moto/moto/core/responses.py", line 136, in _dispatch
    return self.call_action()
  File "…/moto/moto/dynamodb2/responses.py", line 62, in call_action
    response = getattr(self, endpoint)()
  File "…/moto/moto/dynamodb2/responses.py", line 294, in query
    hash_key_value_alias = hash_key_expression.split("=")[1].strip()
IndexError: list index out of range

There is another issue with that code (above), where the order of “AND” and “BETWEEN” (which also has an “AND” in it) matters.

If I remember correctly there’s a TODO in this regard in the code already.

Issue Analytics

  • State:closed
  • Created 8 years ago
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
bblommerscommented, Oct 18, 2019

Hi @TheGhostHuCodes - this test does not work against AWS either, because the ‘begins_with’ operation is not supported against the HASH key. It is supported (both by AWS and moto) against the RANGE key.

I’ve submitted https://github.com/spulec/moto/pull/2491 to ensure moto gives a friendly error message, as opposed to ‘list index out of range’ 😃

0reactions
TheGhostHuCodescommented, Sep 28, 2018

Below is a pytest test that reproduces this issue. Seems to only be an issue when I query using begins_with in the line KeyConditionExpression=Key('key').begins_with('prefix-')).

@mock_dynamodb2
def test_dynamodb():
    dynamodb = boto3.resource(
        'dynamodb',
        region_name='us-east-1',
        aws_access_key_id='MOCK_ACCESS_KEY',
        aws_secret_access_key='MOCK_SECRET_KEY',
    )
    table = dynamodb.create_table(
        TableName='test-table',
        KeySchema=[
            {
                'AttributeName': 'key',
                'KeyType': 'HASH'
            },
        ],
        AttributeDefinitions=[
            {
                'AttributeName': 'key',
                'AttributeType': 'S'
            },
            {
                'AttributeName': 'value',
                'AttributeType': 'S'
            },
        ],
        ProvisionedThroughput={
            'ReadCapacityUnits': 1,
            'WriteCapacityUnits': 1
        })

    items = [
        {
            'key': 'prefix-$LATEST',
            'value': '$LATEST',
        },
        {
            'key': 'prefix-DEV',
            'value': 'DEV',
        },
        {
            'key': 'prefix-PROD',
            'value': 'PROD',
        },
    ]

    with table.batch_writer() as batch:
        for item in items:
            batch.put_item(Item=item)

    table = dynamodb.Table('test-table')
    response = table.query(
        KeyConditionExpression=Key('key').begins_with('prefix-'))
    print(response)
    assert False
Read more comments on GitHub >

github_iconTop Results From Across the Web

Insert into Dynamodb with python giving index out of range error
I am trying to load csv into dynamodb table with python program as below but getting like index out of range error.
Read more >
Python Index Out of Range - Dynamo Forum
Working on a python node and I'm not understanding why I get an index out of range warning here. “IndexError: index out of...
Read more >
how to solve this IndexError list index out of range in python ...
If you are using a loop to access an item, make sure that the loop accounts for the fact that lists are indexed...
Read more >
Querying and scanning an index - Amazon DynamoDB
Compare querying and scanning an index using the SELECT statement in a relational (SQL) database with the Query and Scan operations in Amazon...
Read more >
List Index Out of Range – Python Error [Solved] - freeCodeCamp
In this article, we'll talk about the IndexError: list index out of range error in Python. In each section of the article, I'll...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

s3bucket_path broken from 0.4.19 to 0.4.20

Next page

boto3 + moto + s3 + Docker: "ValueError, "I/O operation on closed file""