Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How put an array of objects at Google Cloud Datastore?

See original GitHub issue

OS : Ubuntu 14.04.5 Python: 3.4.3 google-cloud-datastore: 0.24.0

Code example:

from google.cloud import datastore

class GoogleDatastorePipeline(object):
    
    def __init__(self, settings, stats):
        self.client = datastore.Client()
        self.batch = self.client.batch()

    def open(self):
            self.batch.begin()

    def process(self):
        key = self.client.key('Article')
        entity = datastore.Entity(key)
        entity['name'] = 'A post'
        entity['content'] = '<html></html>'
        # How I insert this as array of objects? Can be indexed, if no, no problem.
        entity['authors'] = [{
            'name': 'Author 1', 
            'type': 'person', 
        },{
            'name': 'Author 2', 
            'type': 'organization', 
        }]

        self.batch.put(entity)

    def close(self):
            self.batch.commit()

Stacktrace:

  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/ubuntu/workspace/ze/pipelines/__init__.py", line 215, in process_item
    self.batch.put(entity)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/batch.py", line 199, in put
    _assign_entity_to_pb(entity_pb, entity)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/batch.py", line 319, in _assign_entity_to_pb
    bare_entity_pb = helpers.entity_to_protobuf(entity)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/helpers.py", line 219, in entity_to_protobuf
    _set_protobuf_value(value_pb, value)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/helpers.py", line 408, in _set_protobuf_value
    attr, val = _pb_attr_value(val)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/helpers.py", line 325, in _pb_attr_value
    raise ValueError("Unknown protobuf attr type %s" % type(val))
ValueError: Unknown protobuf attr type <class 'dict'>
2017-05-03 01:30:16 [scrapy.core.engine] INFO: Closing spider (finished)
2017-05-03 01:30:16 [google_auth_httplib2] DEBUG: Making request: POST https://accounts.google.com/o/oauth2/token
2017-05-03 01:30:16 [scrapy.core.engine] ERROR: Scraper close failure
Traceback (most recent call last):
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/gax/retry.py", line 120, in inner
    return to_call(*args)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/gax/retry.py", line 68, in inner
    return a_func(*updated_args, **kwargs)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/grpc/_channel.py", line 507, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/grpc/_channel.py", line 455, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INVALID_ARGUMENT, Mutation is missing operation.)>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 74, in _catch_remap_gax_error
    yield
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 173, in commit
    return super(GAPICDatastoreAPI, self).commit(*args, **kwargs)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/gapic/datastore/v1/datastore_client.py", line 345, in commit
    return self._commit(request, options)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/gax/api_callable.py", line 419, in inner
    return api_caller(api_call, this_settings, request)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/gax/api_callable.py", line 407, in base_caller
    return api_call(*args)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/gax/api_callable.py", line 368, in inner
    return a_func(*args, **kwargs)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/gax/retry.py", line 126, in inner
    ' classified as transient', exception)
google.gax.errors.RetryError: GaxError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.INVALID_ARGUMENT, Mutation is missing operation.)>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/ubuntu/workspace/ze/pipelines/__init__.py", line 193, in close_spider
    self.batch.commit()
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/batch.py", line 273, in commit
    self._commit()
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/batch.py", line 249, in _commit
    self.project, mode, self._mutations, transaction=self._id)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 173, in commit
    return super(GAPICDatastoreAPI, self).commit(*args, **kwargs)
  File "/usr/lib/python3.4/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 82, in _catch_remap_gax_error
    six.reraise(error_class, new_exc, sys.exc_info()[2])
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 74, in _catch_remap_gax_error
    yield
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 173, in commit
    return super(GAPICDatastoreAPI, self).commit(*args, **kwargs)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/gapic/datastore/v1/datastore_client.py", line 345, in commit
    return self._commit(request, options)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/gax/api_callable.py", line 419, in inner
    return api_caller(api_call, this_settings, request)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/gax/api_callable.py", line 407, in base_caller
    return api_call(*args)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/gax/api_callable.py", line 368, in inner
    return a_func(*args, **kwargs)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/gax/retry.py", line 126, in inner
    ' classified as transient', exception)
google.cloud.exceptions.BadRequest: 400 Mutation is missing operation.

Issue Analytics

State:
Created 6 years ago
Comments:7 (4 by maintainers)

Top GitHub Comments

1reaction

gustavorpscommented, May 3, 2017

I tested the JSON below on Datastore Entites (https://console.cloud.google.com/datastore/entities/new) and worked!

{
    "values": [
        {
            "entityValue": {
                "properties": {
                    "name": {
                        "stringValue": "NAME1"
                    },
                    "type": {
                        "stringValue": "TYPE1"
                    }
                }
            }
        },
        {
            "entityValue": {
                "properties": {
                    "name": {
                        "stringValue": "NAME2"
                    },
                    "type": {
                        "stringValue": "TYPE2"
                    }
                }
            }
        }
    ]
}

screenshot from Google Cloud Datastore Entities

But in python don’t

entity['author'] = {
    'values': [
        {
            'entityValue': {
                'properties': {
                    'name': {
                        'stringValue': 'NAME1'
                    },
                   'type': {
                        'stringValue': 'TYPE1'
                    }
                }
            }
        },{
            'entityValue': {
                'properties': {
                    'name': {
                        'stringValue': 'NAME2'
                    },
                   'type': {
                        'stringValue': 'TYPE2'
                    }
                }
            }
        }
    ]
}

Traceback

Traceback (most recent call last):
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/ubuntu/workspace/ze/pipelines/__init__.py", line 247, in process_item
    self.client.put(entity)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/client.py", line 384, in put
    self.put_multi(entities=[entity])
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/client.py", line 408, in put_multi
    current.put(entity)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/batch.py", line 199, in put
    _assign_entity_to_pb(entity_pb, entity)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/batch.py", line 319, in _assign_entity_to_pb
    bare_entity_pb = helpers.entity_to_protobuf(entity)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/helpers.py", line 219, in entity_to_protobuf
    _set_protobuf_value(value_pb, value)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/helpers.py", line 408, in _set_protobuf_value
    attr, val = _pb_attr_value(val)
  File "/home/ubuntu/workspace/env/lib/python3.4/site-packages/google/cloud/datastore/helpers.py", line 325, in _pb_attr_value
    raise ValueError("Unknown protobuf attr type %s" % type(val))

0reactions

dhermescommented, May 3, 2017

Cheers!

Top Results From Across the Web

Google Cloud Datastore: Storing arrays and objects

You can save a list of Strings in a single property. This property can be either indexed, or unindexed. For an object, you...

Array value types | Cloud Datastore Documentation

Set array value types. ... To learn how to install and use the client library for Datastore mode, see Datastore mode client libraries....

ArrayValue (Cloud Datastore API v1 (Rev. 83) 1.25.0)

An array value. This is the Java data model class that specifies how to parse/serialize into the JSON that is transmitted over HTTP...

Store list in google datastore

I would like to store a ArrayList<Object> in datastore. The values that are supported to be stored in google datastore does not contain...

Google Cloud Storage - TileDB Embedded Docs

For instance, for array gcs://bucket/path/to/array , TileDB creates array schema object gcs://bucket/path/to/array/__array_schema.tdb , fragment metadata object ...