Infinite List Questions, Memory Usage

Hi WatermelonDB developers. Wanted to ask about WatermelonDB usage with an infinite list. I understand that WatermelonDB is lazy first, would I be able to lazily load results of a query until the items are on screen. Wondering if watermelon DB supports something like this using RxJS

$items = $scrollPositionStream.switchMap(postion => runQueryForPosition(position))

Where I have only those objects which are on screen in memory (or those which have appeared on screen once)

[radex]

@Kenneth-KT You’re right, my suggested workaround has real flaws, which depending on the exact use case may be fatal. To be clear: I’m not against adding SQL-level limiting/ordering to WatermelonDB, I just don’t need it for my use cases, so I’m not going to drive the implementation of it. But if this is key to you, I would highly encourage you to consider contributing those, even in a limited form. I can offer guidance. And yes, @azizhk’s PR is another useful workaround for large lists. Using raw SQL queries is also a great option, if you have a good way to know when to trigger re-fetch. Yes, records fetched using SQL are still 🍉 objects and you can observe them normally.

@azizhk

wanted to ask you what is the cache bursting strategy in 🍉?

None. Observable objects are cached on Query, Relation, but only for as long as they’re observed. And Collection has a cache of { id : Record } objects. Once object with given ID is fetched, it’s going to stay there forever.

My initial plan was to implement a two-level cache, keeping Records in a Map for the first X (100? 200?) records, and then move oldest records to a WeakMap, so that they’re cached only as long as some piece of the app actually keeps a reference to it. If memory usage is important to your app, try contributing this improvement - it may help a lot if you’re dealing with a lot of Records

[radex]

Do you think that complete handling is necessary for a PR to be accepted?

No, as long as you clearly convey that the API may change or not fully work, e.g. Q.experimentalSortBy().