Simple chat app cannot be efficiently implemented using Datastore (ex. whatsapp-like app)

Is your feature request related to a problem? Please describe. Hello, considering below syncExpression

syncExpression(Events, (c) =>
  c.meetingId('eq', 'some-value').createdAt('gt', '2020-11-01')
)

Suppose i have a list displaying Events , by above expression only events created after 2020-11-01 will be stored in client-side database.

Now if user keeps scrolling up and suddenly wants to see an event records before 2020-11-01 or the very first oldest event, that data would not be available in client-side.

So how to inform Datastore to sync these older records with backend and display it, because Datastore.query returns only data from client-side .

Also maxRecordsToSync would not help as we dont know in advance the number of older records.

Describe the solution you’d like Something similar to below: when user scrolls to the end of list, call Datastore.queryMore(Events, newLimit) where newLimit is the number of records to sync more from backend. Since Datastore.observe listens to changes in backend database, a similar method Datastore.observeLocal to listen to changes in local database (caused by syncing more newLimit number of records).

newLimit will increase the maxRecordsToSync as maxRecordsToSync + newLimit for either all tables or specified table Events.

Describe alternatives you’ve considered I thought of a solution like when user scroll the end of list then execute Datastore.stop, change the value of maxRecordsToSync in Datastore.config which is usually in some starting file closer toindex.js ( using some hooks in case of React) and then again Datastore.start and finally Datastore.query. But this seems too complicated. I have not tried this though

[mir1198yusuf]

@iartemiev let’s forget everything from discord or above comments. it’s getting complicated.

Just provide a documentation or example app design (only Datastore related thing) for very basic whatsapp-like chat app with just 3 entities - User, Group and Post

(You don’t have to design it completely by yourself. We can discuss each point and design iteratively)

Criteria -

1. User will join by cognito userpool
2. User can create group (so they become admin of that group) or join other group
3. All Users of a group can post messages in group
4. Group admin can delete any one’s post. If any one else than admin deletes it even from local,it should not get deleted at backend.
5. there can be more than one admins. Existing admin can assign other members of group as admins.
6. Admin of one group might be normal user (non admin ) of other group.so he should be able to perform admin function (deleting anyone’s Post ) only in groups where he is admin.
7. Locally, user should have only data of groups he is in and only posts of these groups. Storing other users group or post is security-wise bad practice. User should not be able to query other user data by modifying request from front-end
8. Users can see profile of other users who share atleast one mutual group. Suppose User A and User B dont have any mutual group, they cannot see each other’s profile.
9. User record have a unique field (other than default id field) called as userId. This is basically the sub obtained from Cognito.
10. Users might belong to some groups and might not belong to others
11. Admin of group can delete/remove any user from group. Once user is removed from group, he should be able to send or receive new message to or from that group. So syncing should be near real-time.

Access patterns which will be frequently used :-

1. get single user record
2. get single group record
3. get groups belonging to a particular user
4. get posts belonging to a particular group
5. get single post record
6. get users belonging to a particular group

Design Schemas : - I have created two design schemas for the use case - Relational design and Single table design

Relational schema - User table : id - default PK userId - sub of Cognito - PK of GSI name

Group table : id - default PK - can use this as groupid groupname

GroupUsers table : id - default PK groupId - PK of GSI-A - SK of GSI-B userId - SK of GSI-A - PK of GSI-B

Post table : id - default PK - can use as postId groupId - id of group that the post is created in - PK of GSI createdAt - SK of GSI text - content of post

Single table design - id - default PK - let’s forget this we are keeping but not using it Also,we are using two generic keys - key1 and key2 - two GSI are created on them GSI-A and GSI-B

key1 - PK of GSI-A - SK of GSI-B key2 - PK of GSI-B - SK of GSI-A

For storing user record ; key1 - userId - cognito sub key2 - name (here sort key is not needed but we are storing just to fill value and avoid extra field for name)

For storing group record : key1 - groupId key2 - group name (here sort key is not needed but we are storing just to fill value and avoid extra field for name)

For storing group-users mapping - Remember Groups and Users have m-m relationship key1 - userId key2 - groupId

For storing group-post mapping - Remember groups and post have 1-m relationship key1 - groupId key2 - postId

We will prefix type name before value like User#1233, Group#4344

consider below data according to above mapping - 2 users, 1 group and 2 posts key1 | key2 user#1 | john group#1 | developers user#1 | group#1 group#1 | post#1 group#1 | post#2 user#2 | smith user#2 | group#1

Let’s see how access pattern can work on these 2 GSIs -

1. get single user record - GSI-A : PK as user#1
2. get single group record - GSI-A : PK as group#1
3. get groups belonging to a particular user - GSI-A : PK as user#1 and SK starting with group#
4. get posts belonging to a particular group - GSI-A : PK as group#1 and SK starting with post#
5. get single post record - GSI-B : PK as post#1
6. get users belonging to a particular group - GSI-B : PK as group#1 and SK starting with user#

@iartemiev , cc - @nubpro , @dabit3

Now please specify to satisfy the mentioned criteria - what syncExpression should be (if to use ) or what @ auth directive should be use to protect user data and sync only required data. also where should custom resolvers defined and when it will be enforced ( at syncing time ?)

Use any schema and see if we can implement this using datastore.

If we are able to solve this using Datastore, then it should be added to documentation instead of that very basic notes app.

There is one chat app example on internet but it is a single room or we can say single group chat where all users belong to same group . But in case of whatsapp users might belong to some groups and might not belong to others.

If this is solvable by Datastore, most real world app would be solved. If it fails then I think I have to opt out of Datastore and it means Datastore is still unable to handle such common use cases.

We can deisgn iteratively by discussing on discord .

[iartemiev]

The AppSync team recently posted an RFC detailing their proposal for Enhanced Subscription Filtering. Once this feature is released, DataStore will be able to take advantage of it to enable use cases such as this one.