value_from dynamo query

Currently value_from supports http requests and s3 buckets, but this can cause problems with large files. it would be nice to have the option to store these large files in dynamo and to be able to query it using value_from.

for example

policies:
  - name: vpc-with-dynamodb-value-from
    resource: vpc
    filters:
      - type: value
        key: 'tag:Name'
        value_from:
          url: 'dynamodb:us-east-1:644160558196:table/test'
          format: json
          query: 
            KeyConditionExpression: '#n2 = :v1'
            FilterExpression: 'contains(#n0.#n1, :v0)'
            ExpressionAttributeNames: {'#n0': 'exceptions', '#n1': 'exception1', '#n2': 'test'}
            ExpressionAttributeValues: {':v0': 'testing-vpc', ':v1': '1111111111' }
          expr: '[0].exceptions.exception1'
        op: in

[kapilt]

your aws resources are already available over public http, the difference is protecting the access via IAM for authn/authz, you can do the same with api gw (iam auth).

thats two different native solutions, break up your s3 files to not be full database dumps but policy/env specific values, or front the data with an api gw/http endpoint. for the later we can add support for an auth mode (iam) to sign requests.

[howbazaar]

I’ve been thinking on this, and really just giving an outsider’s perspective to the approach.

I can understand the desire to move from a single large file to particular database queries. However I don’t think that custodian is the right place to provide that functionality.

It seems to me that a simple approach would be to have an api gateway -> lambda that could produce the HTTP response that the value_from filter already understands would be the best apprach. This should be pretty simple to implement, and keeps the database interactions isolated from custodian.

Having DB queries in policies feels like the wrong thing to me.