[DataStore] CursorWindowAllocationException crash for large datasets

Hey guys, I have some issues and usage questions:

What I am doing is just calling the first query method on the DataStore of a single model, and at that moment Amplify starts to sync/download all the data from all the models.

Issue description:

The problem I noticed is that when the amount of data in the DynamoDB started to increase, the first synchronization started to fail.

We have two main models, Location and Activities. Each location has N activities and they will be assigned to each user on a daily basis.

We made some calculations and found that each user will have approximately 1350 activities assigned per month within 150 different locations. We want to previously populate the database with locations and activities for a full month for each user. So each month will have 1500 rows in total. With this amount of data, I found the following issues just in the first sync:

First sync: When the user logs-in for the first time and I perform the first query, after that everything seems to work fine.

First query:

fun getTaskList(
        onResponse: (value: List<Ubicacion>) -> Unit,
        onFailure: (value: DataStoreException) -> Unit
    ) {
        Amplify.DataStore.query(Ubicacion::class.java,
            Where.matches(Ubicacion.FECHA.eq(Utils.getCurrentDateString())),
            { matches ->
                Log.i(
                    "TaskDataStore",
                    "Obtained today's ${Utils.getCurrentDateString()} task list: $matches"
                )
                onResponse(matches.asSequence().toList())
            },
            { onFailure(it) }
        )
 }

Issues:

1. In low-range smartphones (Samsung Galaxy A8, A10 & LGE LM-K410), I found this issue:

CursorWindowAllocationException
Cursor window allocation of 2097152 bytes failed. # Open Cursors=442 (# cursors opened by this proc=442)

• More information in: https://sentry.io/share/issue/78f9af09c61b4a6e925035f6819ae6cd/

1. In mid-range smartphones, sometimes I have a TimeOutException like in this issue: #563

Expected result:

Amplify only syncs/download the locations and activities of this single day, for this user. So the app doesn’t have memory issues. OR Amplify syncs/downloads by small chunks of data.

Current result

Amplify syncs/downloads all the information previously registered information and in low-mid-range smartphones the application crashes, even though I only queried today’s locations.

Usage question:

Is there a way to implement this with the current version of the API or this would be something like a new feature?

[jamesonwilliams]

Hi @upachecog!

calling the first query method on the DataStore of a single model, and at that moment Amplify starts to sync/download all the data from all the models.

As of the last few versions of the DataStore, this is the expected behavior. Whenever you invoke a DataStore API, if the sync engine is not already running, it will be started.

Expected result: Amplify only syncs/download the locations and activities of this single day, for this user.

@richardmcclellan is implementing a featured called “selective sync,” right now, which should address this problem. With it, you would be able to define filters to select the data returned by the DataStore’s initial synchronization calls. In your case, you could leverage a timestamp of some kind, to only get recent data. This work is already complete for the JavaScript library, and you can see it here: https://github.com/aws-amplify/amplify-js/pull/7001.

[upachecog]

@richardmcclellan Thanks for your answer and help.

• Now I am using version 1.6.2 and I’ve implemented selective sync.

I have a couple of questions, though:

1. I want to sync only the Locations I have to visit today, so I added this syncExpression:

.syncExpression(Location::class.java) {
     Location.DATE.eq(Utils.getCurrentDateString())
}

The question is: How can I filter my Activities based on the first query if I don’t have a Date field in the Activity model but I have a @connection to the Location Model?

Based on the next query I found here:

Amplify.DataStore.query(Comment::class.java, Where.matches(Post.STATUS.eq(PostStatus.ACTIVE)),
    {
        while (it.hasNext()) {
            val comment: Comment = it.next()
            val content: String = comment.content
            Log.i("MyAmplifyApp", "Content: $content")
        }
    },
    { Log.e("MyAmplifyApp", "Query failed.", it) }
)

I tried to do something like the following, but of course it doesn’t work:

.syncExpression(Activity::class.java) { 
    Location.DATE.eq(Utils.getCurrentDateString())
}

1. There are some other models which I don’t need to sync, actually I only use mutations there. The question is: Using selective sync, can I completely disable synchronization to these models?