[proposal] new Storage API

Is your feature request related to a problem? Please describe. Veramo can currently use a single data source for credentials, presentations and messages; an implementation of the IDataStore and IDataStoreORM interfaces.

Most other veramo top-level plugins use a layered approach, where the top-level plugin acts as a coordinator between multiple lower level implementations of a common interface.

Also, the IDataStoreORM interface is based on a relational data model, with a lot of assumptions about the connections between the data items being stored. This makes new implementations difficult, so a new solution should be adopted.

Describe the solution you’d like A new storage model should support multiple data sources. Adopting such a pattern for storage, would bring it in line with the rest of the API as well as allowing users to store data in multiple locations (local private data, remote backup, remote inbox service, remote public information)

The solution chosen should be able to run queries on the data. Examples of common queries include:

• all credentials issued by an issuer
• issued after a certain date
• containing a certain claim
• matching a certain Type
• all messages of a particular type
• all presentations that include a certain @context
• a VC/VP/message by ID
• …

Equally important is the ability to filter for credentials using JSONPath matches that are used in the DIF Presentation Exchange and Credential Manifest protocols.

Additional context

Some related projects already use an adapter pattern to support multiple sources:

• verida data-store
• blockchain labs:um VCManager

[andyv09]

We’re currently upgrading and updating VCManager for our Snap. This is what we’ve come up with so far.

Idea is to keep structure similar to the one we’ve used before, and we think covers all the use cases mentioned above.

A couple of notes:

• Filter can be selected (JSONPath, etc.). Most use cases should use JSONPath, but we still wanted to leave room for improvement
• Filtering will be done in VCStorePlugin to keep VCManager as lightweight as possible
• id used in delete is a sha256 hash of VC as id is not mandatory in every VC
• save can save the same VC in one or more different stores (e.g. save(vc, [‘database’, ‘google_drive’])
• query will return all VCs if filter and store are not provided
• Function clear can be added to remove all VCs from selected store

[mirceanis]

Thank you for the update, this is great!

I really like the fact that queries are propagated to each implementation because this would allow folks to run the query where the data lives instead of centralizing it locally.

I have a few questions:

• is the filter type defining the type of query that follows? You mention "JSONPath", but would this be something like "couchdb", or "SQL" in other cases?
  • if so, did you also sketch any error scenarios for when a VCStorePlugin implementation cannot implement a type of query, or when the query fails for some exceptional reason?
• How about storing something else than VCs? like presentations or messages, or anything else that looks like JSON and potentially has an id. It looks like the interface you defined is not limiting to VCs, which is great, IMO.

And suggestions:

• The result of query() could be a list of objects containing data+metadata instead of using a returnStore parameter to select the return type. For example:

[
   {data: {<W3CVC>}, meta: {store: "local", id: "asdf"}},
   {data: {<W3CVP>}, meta: {store: "remote", id: "fghj"}
]

• All methods of VCStorePlugin should support an options?: {} parameter for easy customization without interface changes. Something like this could be then used to forward authorization parameters for data stores that require them.

---

Please correct me if I misunderstood the API you described for filtering