Query Support Proposal

Opening this in it’s own ticket since it’s sort of long and I wanted to allow it to be closed separately in the case I am completely insane.

I’ve been tinkering with this library for the last couple of days and trying to imagine how queries could work. A few things strike me as having potential here.

Model Introspection
First, the array definition for composite keys potentially gives the model some important information.For example, if only 1 of 2 parts of the composite exist, we may be able to infer that the query needs to have a begins_with() on the SK.

Potentially the GSI in use could be inferred by which data items are populated as well, though there may be a need to provide an explicit hint in some cases.

GSI Formatting
All GSI pk and sk schema formatting could probably follow the same schema formatting rules as the main pk and sk do. They could be dynamically constructed just like the main table SK constructed. The model would just need to flag those schema elements as belonging to a GSI. This could be done on the schema element itself, or in the main model definition.

As a side note, I think the gsi and table definition data could be lifted out of the model and defined separately for easier single table design.

DAT 403 Example
I’m a huge fan of Rick Houlihan’s ReInvent presentations, as I know @jeremydaly is as well. I just spent a moment trying to envision how the employee model from his 2019 presentation might be constructed and queried. Here is what I came up with. There are likely errors

const Employee = new Model('Employee', {

  // Specify table name
  table: 'singletable',

  // Define partition and sort keys
  partitionKey: 'pk',
  sortKey: 'sk',

  // GSI here - type: gsi | local, defaults to gsi
  indexes: [
    { name: 'gsi1', partitionKey: 'gsi1pk', sortKey: 'gsi1sk' },
    { name: 'gsi2', partitionKey: 'gsi2pk', sortKey: 'gsi2sk' },
    { name: 'gsi3', partitionKey: 'gsi3pk', sortKey: 'gsi3sk' }
  ],

  // Define schema
  schema: {
    // primary keys
    pk: { type: 'string', default: (data) => `${data.employeeid}` },                    // Access Pattern 9
    sk: { type: 'string', hidden: true },
    sk0: ['sk', 0, { type: 'string', default: (data) => `${data.type}` }],              // Access Pattern 9
    sk1: ['sk', 1, { type: 'string', default: (data) => `${data.managerid}` }], 

    // GSI1
    gsi1pk: { type: 'string', default: (data) => `${data.employeeemail}` },             // Access Pattern 10
    gsi1sk: { type: 'string', default: (data) => `${data.sk}` },                        // Access Pattern 10

    // GSI2
    gsi2pk: { type: 'string', default: (data) => `${data.manageremail}` },              // Access pattern 15
    gsi2sk: { type: 'string', default: (data) => `${data.sk}` },                        // Access pattern 15

    // GSI3
    gsi3pk: { type: 'string', default: (data) => `${data.city}` },                      // Access pattern 14
    gsi3sk: { type: 'string', hidden: true },                                           // Access pattern 14
    gsi3sk0: ['gsi3sk', 0, { type: 'string', default: (data) => `${data.building}` }],  // Access Pattern 14
    gsi3sk1: ['gsi3sk', 1, { type: 'string', default: (data) => `${data.floor}` }],     // Access Pattern 14
    gsi3sk2: ['gsi3sk', 2, { type: 'string', default: (data) => `${data.aisle}` }],     // Access Pattern 14
    gsi3sk3: ['gsi3sk', 3, { type: 'string', default: (data) => `${data.desk}` }],      // Access Pattern 14

    // data
    type: { type: 'string', default: 'E' },
    employeeid: { type: 'string' },
    employeeemail: { type: 'string' },
    hiredate: { type: 'string' },
    managerid: { type: 'string' },
    manageremail: { type: 'string' },
    city: { type: 'string' },
    building: { type: 'string' },
    floor: { type: 'string' },
    aisle: { type: 'string' },
    desk: { type: 'string' }
    // ... other data attributes

  }
})

Then you could query the access patterns thusly:

Access Pattern 9 - Employee By ID Since type maps to the first part of the composite sk, we can infer begins_with(). The main table’s pk and sk are populated but none of the GSI’s are, so we can also infer that this is a query against the table and not a GSI.

let item = {
  employeeid: 'employee22',
  type: 'E' 
}
let params = Employee.query(item)

Access Pattern 10 -Employee by Email This is similar to the above example but the main table’s pk will not be populated. Instead, GSI1’s pk and sk will be populated, but no others, so we can infer that this query should be against GSI1.

let item = {
  employeeemail: '[e-mail]',
  type: 'E'
}
let params = Employee.query(item)

Access Pattern 14 - employees by city, building, floor, aisle, desk This one populates parts of GSI3, along with a partial gsi3sk, so we can infer that this query will be against gsi3, and will require a begins_with() on gsi3sk.

let item = {
  city: 'Atlanta',
  building: 'Davis West',
  floor: '4' 
}
let params = Employee.query(item)

Access Pattern 15 - Employees by Manager This one is a little fuzzy because I am not entirely certain which elements in Rick’s table contained the managerid - maybe it’s the second element in the table’s sk? I’m just guessing. But if that’s true, this query will grab the employees that this person manages, again inferring gsi2 and begins_with() on gsi1sk.

let item = {
  manageremail: '[e-mail]',
  type: 'E' 
}
let params = Employee.query(item)

There it is, warts and all. Hopefully at least some of this will add to the conversation.

[jeremydaly]

Queries are now supported on the v0.2 branch.

[jeremydaly]

Not yet, @clarkd. But you can just run npm install jeremydaly/dynamodb-toolbox#v0.2 to install v0.2 as a dependency.