Can't store entities with embedded entities which have properties > 1500 bytes

Seems that excluding a top level property doesn’t exclude an embedded entities properties from being indexed - on top of that, I don’t believe there’s a way to exclude an embedded entities individual properties, that would solve my problem too.

Steps to reproduce

Just use this script (I’m running the datastore emulator)

const Datastore = require('@google-cloud/datastore');
const datastore = new Datastore({ projectId: 'datastore-test' });

datastore.insert({
  key: datastore.key('User'),
  data: [
    {
      name: 'description',
      value: {
        text: Buffer.alloc(1501, 'a').toString(),
      },
      excludeFromIndexes: true
    }
  ]
}, (err, result) => {
  if (err) {
    console.error(err);
  } else {
    console.log(result.data);
  }
});

I’d expect there to be no error - the embedded entity ({ text: '...' }) shouldn’t be indexed I believe? But it throw an error saying:

Error: The value of property "text" is longer than 1500 bytes.

Environment

• OS: OS X Sierra
• Node.js: v6.7.0
• npm: 3.10.3
• @google-cloud/datastore: v0.6.0

[ggprod]

Could I ask what the status of this issue is? I have some entities that have embedded entities that have properties that are >1500 Bytes. When I try setting excludeFromIndexes: true on the embedded entity I still get an error thrown for its sub-properties that are >1500 Bytes. Is there some way I can store them?

[bedeoverend]

@stephenplusplus thanks for the options - I think the data seems to make the more sense to me… i.e. value is for more ‘basic’ types, and data for embedded objects.

It’s pretty fringe edge case at this point - and I’m not sure if it’s even possible to do, but if you had an array of objects, could you index them?

e.g.

var rawPersonEntity = {
  key: datastore.key('Person'),
  data: [
    {
      name: 'name',
      value: 'Mary'
    },
    {
      name: 'metadata',
      data: [
        {
          excludeFromIndexes: true,
          name: 'bio',
          value: '...'
        }
      ]
    },
    {
      name: 'subjects',
      value: [
        {
          excludeFromIndexes: true, // Is this something that can be done? Is there a reason why it would be done?
          name: 'history',
          credits: 2
        },
        // ...
      ]
    }
  ]
}

Other than that, I can’t see any issue with them…Another option would be making an datastore.value style factory - like datastore.key - where you can specify types and the data associated with it e.g.

datastore.insert({
  key: datastore.key('Person'),
  data: datastore.value({
    type: datastore.Entity, // Defaults to 'auto'
    data: {
      name: datastore.value({
        type: String,
        value: 'Mary'
      }),
      metadata: datastore.value({
        type: datastore.Entity,
        value: {
          bio: datastore.value({
            type: String,
            excludeFromIndexes: true,
            value: '...'
          })
        }
      }),
      subjects: datastore.value({
        type: Array,
        value: [
          datastore.value({
            type: datastore.Entity,
            value: {
              name: datastore.value({
                type: String,
                value: 'history',
              }),
              credits: datastore.value({
                type: Number,
                value: 2
              })
            }
          },
          datastore.value({
            type: datastore.Entity,
            value: {
              name: datastore.value({
                type: String,
                value: 'french',
              }),
              credits: datastore.value({
                type: Number,
                value: 4
              })
            }
          },
          datastore.value({
            type: datastore.Entity,
            value: {
              name: datastore.value({
                type: String,
                value: 'maths',
              }),
              credits: datastore.value({
                type: Number,
                value: 4
              })
            }
          }
        ]
      })
    }
  });

Ha - after writing that all out, I realise how incredibly over the top that looks, but it feels robust. Perhaps not as a candidate for quickly writing out simple entities, but perhaps as a way that will work well with a helper utility? e.g. for my use case, I’m converting incoming JSON to an entity, and I want to recursively step through it and add excludeFromIndexes as needed…thoughts?