Stuck on an issue?

We’re currently facing quite a significant performance issue with Slickgrid when doing a lot of updates in the DataView.

From a data stream we’re receiving thousands of log entries, which we want to feed into slickgrid via a DataView. As part these incoming events we need to delete some and add some new ones. The most nasty thing we might be doing is inserting all items at the beginning of the list because we want to have newest items first.

We’re calling beginUpdate and endUpdate but this only suspends the refresh triggers. Any call to insertItem, addItem or deleteItem causes the updateIdxById to be called which in worst case causes a loop over all items again.

So assume you have a dataview with 3000 items, and you get 300 new items which will either call insertItem(0, item) or deleteItem(item.id) based on some rules. In the worst case we will have 300 times a loop over 3000 items to update the index table. As the updateIdxById and idxById are not public, it is really hardly possible to optimize here on our level with some known constraints.

The insert part I was able to patch already by modifying the original items array and calling insertItem at the very end:

dataView.beginUpdate();
for(let i = 0; i < logs.length; i++) {
    if(i < logs.length - 1) { dataView.getItems().unshift(item); } 
    else { dataView.insertItem(0, item); }
}
dataView.endUpdate();

This way I ensure that we have 1 loop over 3000 items at the last item. But it gets tricky if you start deleting items because the internal index table is not correct anymore and also delete calls cannot easily be translated.

dataView.beginUpdate();
for(let i = 0; i < logs.length; i++) {
    if (shouldRemove(item)) { dataView.deleteItem(item.id); }
    else if (i < logs.length - 1) { dataView.getItems().unshift(item); } 
    else { dataView.insertItem(0, item); }
}
dataView.endUpdate();

My idea on how SlickGrid could handle this would be something like this:

1. We add something like a beginBulkUpdate() and call it before the update.
2. Add/Insert operations are done instantly and the internal items array is changed, the id to index mapping is NOT updated yet.
3. The delete operations are remembered in a separate Set instance (or as object key if you want to support old browsers).
4. At the end we call an endBulkUpdate() which loops once through the whole dataset and either update the index of the current element, or delete the item at the current location if it is contained in the Set

I would have added this to my level of the code but the as the index loop is private I cannot enforce this properly.

It would be great if the DataView could handle such kind of add or update operations. SlickGrid in general is operating fine, it’s the streaming update capabilities which are not available. Reimplementing various functionalities of the DataView we need is also not really a good option.

Alternatively I would be also fine with just exposing the currently private parts of the lookups to allow me changing it. This way I can load the original list and idx lookup. This way I should be able to insert new items and delete old ones. Finally I can just call setItems() with the final list or update idxById on my own.

Looking forward to your feedback.