[Epic] Simplify CRUD Queries v1

Objective

Reduce the time to ship useful plugins to 1 week. Today it takes 8 weeks to get plugins into a usable state. While the initial plugin does come out fairly quickly, it usually requires a lot of iteration to become useful. There are 2 core gaps here

• Lack of clarity on what features should the plugin support
• Poor UX on the plugin which makes it unusable without documentation

Key Points

• Standardise Plugin Requirements for all CRUD operations
• A design system and guidelines so we can add high-quality integrations with as little design as possible.
• Components for complex interactions like array object input.
• A solid contributor’s guide to integrations by September 2021, so that we can scale integrations during Hacktoberfest.

Success Metrics

Goal	Metric
High-Quality DB Integrations	15 DB Integrations that anyone can use without documentation
High-Quality SAAS Integrations	10 SAAS Integrations that anyone can use without documentation
Simple Contribution Platform	Get 5 external contributions

Requirements

Requirement	User Story	Importance	Notes
Unified Interface Generator	Our system should be able to generate the interface for database/SAAS integrations with minimal input from the contributor	High
Intuitive Interface	A user should have a homogenous intuitive experience using integrations on appsmith	High
Homogenous DB Functionality	A user should be able to achieve the same operations across DB integrations	High
Raw Mode	A user should be able to access the underlying query construct and make use of it with the aid of documentation.	High
SAAS Ingestion App	A contributor should be easily able to add a SAAS integration without help from the appsmith team	High
DB Contribution Interface	A user should have a simple interface with documentation to contribute a new database integration	Medium

Out of Scope

ORM Method of querying data

Unified Interface Solution

Databases are generally structured with homogenous entities i.e Tables, Collections, Files, etc and these are the entities on top of which operations are generally performed. DB Interfaces can be modeled around these entities to simplify the querying experience for users. DBs are also largely used for CRUD and thus the number of operations to be performed on them can also be limited to Create, Read, Update & Delete. For our existing databases, this could look like

1. Users must select a DB (Google Sheet, Firebase, SQL, Mongo, S3, Elastic, Redis, Dynamo, Big Query)
2. A user must select an operation
   • Fetch
   • Insert
   • Update (Not valid for S3)
   • Delete
   • Aggregate (Not valid for S3)
   • Bulk Insert
   • Bulk Update
   • Bulk Delete
   • Raw Query (Not valid for Firebase / S3)
   • Custom Action (?)
3. A user must select an entity / sub-entity (optional) / ID (optional) in the DB. They can be configured Dynamically as well.
   • Google Sheets (SpreadSheet)
     • Sheet Name (Optional)
       • Table Header Index
         • Row Index
   • SQL (Table)
     • Primary Key
   • Mongo/Firestore/Firebase (Collection)
     • Object ID (without object Id syntax)
   • Redis (DB)
     • Key
   • S3 (Bucket)
     • File (Optional) (Only valid for Fetch Op)
   • Elastic (Index)
     • Document (Optional)
   • Big Query (Projects)
     • Datasets (Optional)
       • Models (Optional)
       • Routines (Optional)
       • Tables (Optional)
     • Jobs (Optional)
     • Queries (Optional)
   • SnowFlake (Table)

Fetch

This operation is used to fetch the data inside the entity. This operation requires a user to configure

1. Where Clause: This is a UI element that allows the user to configure multiple conditions to fetch the data. The values can be configured dynamically. if the plugin does not support this, the server should do the filtering.
2. Limit: This is a UI element that allows a user to configure the number of rows of the entity to return. If the plugin does not support this, the server should limit the amount of data returned. A default value must be present for every plugin to prevent excessive data from being returned.
3. Offset: This is a UI element that allows users to configure how many records to skip from the first record that matches the where clause. The default value of this field should be 0.
4. OrderBy: This is a UI element that selects the column(s) by which to order the data. If the plugin does not support filtering server-side, the server should order the data. It should also allow users to select an ascending / descending order
5. Projection: This is a UI element that selects which fields of the response to return. This field can be hidden for single key responses like S3 files.
6. Options: This is a UI section that contains plugin-specific additional parameters such as
   • Enable URL Signing (S3)
   • URL Expiry Duration (S3)

The response of this operation should always be an array of objects with keys and values.

Insert

This operation is used to insert a new row into the entity. This operation requires a user to enter an object. The user should be made aware of the object format and have a sample object to refer to.

The response of this operation should be a JSON of the form

{
"message": "Row successfully Inserted"
}

Update

This operation is used to update a row in the entity. This operation requires a user to enter 3 fields

1. New Object properties:
2. Update Diffs: This switch will update only the keys specified for JSON objects. This is enabled by default. If this is disabled, unspecified keys will be deleted from the object. This field should have a tooltip explaining how it works

The response of this operation should be a JSON of the form

{
"message": "Row successfully updated"
}

Delete

This operation is used to delete a row in the entity. This operation requires a user to enter 3 fields

{
"message": "Row successfully deleted"
}

Bulk Insert

This operation is used to insert a new row into the entity. This operation requires a user to enter an array of objects. The user should be made aware of the object format and have a sample object to refer to. The response of this operation should be a JSON of the form

{
"message": "10 Rows successfully Inserted"
}

Bulk Update

This operation is used to update multiple rows in the entity. This operation requires a user to enter the following fields

1. Update Type:
   • Where Update:
     • Filter: This section should contain the where / limit / offset / order by sections in the Fetch OP.
     • New Object properties: This operation requires a user to enter an object. The user should be made aware of the object format and have a sample object to refer to.
   • Objects Update: This operation requires a user to enter an array of objects. The user should be made aware of the object format and have a sample object to refer to. Each object of the array has an Id field.
2. Update Differences: This property will update only the keys specified for JSON objects. This is enabled by default. If this is disabled, unspecified keys will be deleted from the object.

The response of this operation should be a JSON of the form

{
"message": "10 Rows successfully updated"
}

Bulk Delete

This operation is used to delete multiple rows in the entity. This operation requires a user to enter 2 fields.

• Filter: This section should contain the where / limit / offset /order by sections in the Fetch OP.

{
"message": "10 Rows successfully deleted"
}

Aggregate

This operation is used to return aggregate data of results. The backend will do the aggregation logic for plugins that do not support it.

1. Operator: The user can select from the following operations AVG | MIN | MAX | COUNT | DISTINCT | SUM
2. Column: The user can select a key of the entity to apply the operation on
3. GroupBy: The user can select a key of the entity to group the response operation by

The response of this operation should always be an array of objects with keys and values.

Design Components Required

1. Operation Selector
2. Entity / Sub Entity / ID Selector
3. Where Filtering Section
4. Insert / Update Body
5. Bulk Insert / Update Body
6. Projection Selector
7. Options

DB Contribution Solution

SAAS Ingestion Solution

Steps required for a new integration are recorded here.

Figma

https://www.figma.com/file/5x2k7Uz4TKHRqKPtdqgsBB/Unified-Query-Interface?node-id=118%3A2750

[areyabhishek]

I have a recommendation here because it seems like we’ll now be using forms at a lot of places. If we are using forms, always expose the entire form data as a JSON so that it’s easy to copy paste across queries/apps/orgs, etc. The advantage of form is that you can input things quickly, but they make copy/paste difficult. So let’s make it a default option to always be able to access the JSON. It’s a small feature that’ll rapidly improve the DX.

[prapullac]

Closing the issue