How to lazily/dynamically fetch data if the number of rows is known in advance

My use-case

I need to display a large amount of rows (upwards of 300k) where the number of rows is precalculated on the backend and passed to the frontend but where it’s not viable to prefetch all the data at once before rendering the table. The backend will churn out chunks of about 2k dataItems for each request and even that takes a few seconds. Furthermore small changes to available filters will result in a completely different number of items so again prefetching and caching all the data in advance unfortunately is not an option. Paging too has been dismissed as an option given the usability drawbacks we’ve encountered so far with our existing solution.

The Problem

I am having trouble coming up with an elegant solution (or the right way) to lazily/dynamically fetch the row-data once they are required (the user has scrolled those rows into view). From what I understand, react-base-table requires a data aray and queries both the length and the row-data from that property. There is also dataGetter for individual cells but as far as I can see, it too relies on an existing entry for the row-data in question. I see no easy way to tell react-base-table that there are n rows but have react-base-table query the data using a custom getter function.

My current approach

Currently (lacking better ideas) the approach I would pursue is to pass a proxy object to react-base-table via the data property and return the length of the available data (which is known to us as that has been precalculated by the backend as outlined above). I would then override the length attribute and also provide a placeholder object when data for a row is queried that has not yet been fetched. I would also schedule the fetching of the missing row data using the parameters passed to onRowsRendered and once the data has been fetched force an update using forceTableUpdate.

Question

Is there a better approach?

[nihgwu]

hey @momesana thanks for your inspection, I just checked locally and you are right that the PropTypes will iterate the data, if I change data: PropTypes.arrayOf(PropTypes.object).isRequired to data: PropTypes.array.isRequired, that would be OK, and if you are fine with that I’m happy to make the change

I’d like to mention that the prop types check will be disabled in production

BTW, on my MBP it costs 0.03s to create 300k items via const data = Array.from({ length: 300000 }, (_, idx) => ({ id: idx })) or 0.13s via const data = Array.from({ length: 300000 }, (_, idx) => ({ id: idx, ...shared })), I used skeletonItem to make sure dataKey="a.b.c" work out of box, but if you don’t have those kind of dataKey or you use dataGetter, you just need to initialize with ({ id: idx }) without skeleton item

And if the total height of the rows exceeds a certain value see #143 , like 400000 items with rowHeight=50, when you scroll to the end, that’s a know issue for react-window and the author doesn’t want to fix(there is workaround in react-virtualized)

[momesana]

hey @momesana thanks for your inspection, I just checked locally and you are right that the PropTypes will iterate the data, if I change data: PropTypes.arrayOf(PropTypes.object).isRequired to data: PropTypes.array.isRequired, that would be OK, and if you are fine with that I’m happy to make the change

I’d like to mention that the prop types check will be disabled in production

BTW, on my MBP it costs 0.03s to create 300k items via const data = Array.from({ length: 300000 }, (_, idx) => ({ id: idx })) or 0.13s via const data = Array.from({ length: 300000 }, (_, idx) => ({ id: idx, ...shared })), I used skeletonItem to make sure dataKey="a.b.c" work out of box, but if you don’t have those kind of dataKey or you use dataGetter, you just need to initialize with ({ id: idx }) without skeleton item

And if the total height of the rows exceeds a certain value see #143 , like 400000 items with rowHeight=50, when you scroll to the end, that’s a know issue for react-window and the author doesn’t want to fix(there is workaround in react-virtualized)

Hi @nihgwu, first of all thank you very much for taking the time. I really appreciate it. If changing the prop-type to just array prevents the inspection of every item that would be awesome. I must confess, I didn’t test it in production yet but I suppose I would rather have it not iterate through all the items since I’ll have to test it with large lists in development due to the special use case of our customers.

As for the performance, I am using a lenovo T580 with an i7. The performance also largely depends on the JavaScript engine in use and older versions are less optimized and slower. They also usually tend to be installed on older machines which doesn’t really help make things better. I think the approach of having something like a “store” and being able to query the size and items from the store on the demand (which is the approach introduced by the ExtJs data grids) is more flexible than having an array but that’s also something I could live with.

Regarding the height issue or rather the number of items, it’s something that most libraries suffer from (again the ExtJs data grid is an exception and I am not sure how they did it). I once implemented a simple virtualized data-grid and ran into the issue as well upwards of a few hundred thousand items. The limit actually depends on the row-height. The issue (and thus the number of “reachable” items depends on the implementation specifics of the browser and is caused by the limits regarding the maximum possible height of an element in pixels (https://stackoverflow.com/questions/16637530/whats-the-maximum-pixel-value-of-css-width-and-height-properties) and how it interacts with the values and resolution of the scrollbar. You can bypass that if you don’t use the native scrollbar and instead implement all of that from scratch but that means a lot of work and a lot of edge cases to take care of and an impact on performance.Therefore, I can sort of understand why the react-window maintainer refuses to implement it.

Again thanks a lot in advance for the Prop-Types fix + the hint about the prop-types checks being disabled in production. I am Looking forward to using it.