open new tab (`t`) dialog very slow to load suggestions

Error details

SurfingKeys: Version 0.9.66 from AMO

Browser: Firefox 78.0.2 (64-bit) Linux.

URL: All urls

Context

On any page, pressing t with the default bindings brings up the in-page location bar near instantly. However, typing any letter results in a few seconds long pause and CPU usage spike while before it loads any suggestions, making the t functionality in SurfingKeys near unusable compared to the native location bar. Even once the initial suggestions load, typing any additional letters or hitting tab makes the browser unresponsive for a few seconds.

This is on my default firefox profile with plenty of other extensions and a large amount of history/bookmarks. I haven’t tried bisecting my extensions or settings yet to see if I can eliminate the slowdown. However, the similar functionality of vimium’s “omnibar” worked fine, as does the native firefox bar, so I suspect something with surfingkey’s history loading specifically makes the process slow/expensive.

[perfectayush]

Okay, I built up on the above performance analysis, this took a while.

Due to multiprocessing in Firefox, the extension loads background.js in another process and some other parts like omnibar.js in the tab process. This also means that performance recording of both need to be analyzed in 2 different inspectors.

After doing that analysis properly, I realized that having a very big history causes slowdown issues with SurfingKeys omnibar in Firefox. The slowdown is mainly in 2 places:

1. function _getHistory in background.js: This function calls a sort when sortByMostUsed is True. The sort in firefox which is implemented as Promise. When number of history items are large, around 85k in my case, there is a significant slowdown in this function. I tested by limiting history items to 2000 in the function using maxResults argument of chrome.history.search.
2. function _filterByTitleOrUrl in omnibar.js. This function calls filter over all the urls using regex generated by input string query in omnibar. Again this filter is significantly slowed when the url item are quite large.

To fix this issue, we will require to come up with strategy to speed up this sort and filter in these 2 functions. A couple of quick hacks, I could think of are:

1. To limit the items returned by _getHistory in some fashion, this will solve issue for filter in omnibar.js. Making this limit configurable. This however can lead to weird results as complete history won’t be loaded.
2. Deleting browsing history is another not so great option. Truncating/Archiving it in some fashion is another similar option.

[jahkeup]

To limit the items returned by _getHistory in some fashion, this will solve issue for filter in omnibar.js. Making this limit configurable. This however can lead to weird results as complete history won’t be loaded.

I wonder if icremental loads could happen after exhausting the current list of results - maybe either asynchronously after reaching some threshold (only if results come already sorted by MRU) or by spawning another query after the user pauses. Upon receiving the results from either kind of query, the results could be merged in the list of offered results or replace the existing items entirely (with care to not disrupt a user’s selection).

There’s probably also a hybrid approach as well where a very small list (again, MRU) is initially searched while another background query resolves “the rest” of the results from deeper history.

I’m not familiar with the code and prior issues made, so forgive me if this is asked/answered/impossible: is it possible to cache the history items, and periodically refresh, centrally? And incrementally fill in the deeper wells of history over time?