question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Performance issue with large list and long search text

See original GitHub issue

I’m not sure am I doing something wrong, but I noticed that when doing search with a very long string and quite low threshold of 0.3, the search becomes very slow ~8-900ms with MBP 2015.

It’s hard for me to pinpoint exactly the cause, but since there isn’t issues searching with a short string, I guess the longer string just isn’t handled well by a computation, that isn’t exactly O(1).

https://imgur.com/a/T9OYGJ1

That’s the lowest level where my Chrome JS profiler was able to get to. Doing simple performance check with:

      console.time('search')
      const newResults = fuse.search(newVal)
      console.timeEnd('search')

confirms that this indeed is the culprit.

A value for newVal might be eg Ajoneuvot ja liikenne / Autot / Automerkit / Volkswagen / Bora, Vento ja Jett. Fuse is provided a list, where the first searched value is string (similar to that previous one), the second the splitted version of that same string from ‘/’ eg ['Ajoneuvot ja liikenne', 'Autot', 'Automerkit', 'Volkswagen', 'Bora, Vento ja Jett']. The options object being: {threshold: 0.3, keys: ['text', 'breadcrumb'], maxPatternLength: 32 }.

My search items list is 2427 long, with a single instance being eg:

  {
    breadcrumb: ["Ajanviete"]
    key: 52
    text: "Ajanviete"
    value: 52
  }

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:1
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
TeemuKoivistocommented, Nov 22, 2019

I made a demo here https://teemukoivisto.github.io/search-demo-app/

Quite weirdly, it is much faster when I have Chrome’s developer tools hidden. Now it’s even usable on my Android. But it’s still laggy and the worst of all, when I have a specific topic chosen, topics close to it don’t show in sequential fashion.

Eg I have topic Harrastukset / Pelit / Vedonlyönti ja rahapelit / Pokeri / Pokeriturnaukset chosen, yet after first 10 or so items I start getting topics from way off the original eg Viihde ja kulttuuri / Kirjallisuus / Sarjakuvat / Omat sarjakuvat. It should be showing topics from the same parent topic eg Harrastukset / Pelit / Vedonlyönti ja rahapelit / Yleistä vedonlyönnistä and work its way down. I added the tokenize: true parameter but it didn’t help.

I guess this means I have to make my own custom search. I had hoped I would avoid using dull substring search but I guess I have to make use of it. Maybe use only fuzzy search for the last 10 characters of the string.

0reactions
peteroidcommented, Feb 7, 2021

I am facing the same issue as on my React Native app. There are 2k items and the performance significantly dropped when the search text was growing from 1 to 100 chars.

Setting the threshold to lower value (lower fuzziness) would help a bit.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Why Full Text's CONTAINS Queries Are So Slow
When your query uses CONTAINS, SQL Server has a nasty habit of doing a full text search across all of the rows in...
Read more >
python performance issue while searching in a huge list
I need to speed up (dramatically) the search in a "huge" single dimension list of unsigned values. The list has 389.114 elements, ...
Read more >
Improve the Performance of Full-Text Queries - SQL Server
The following list of recommendations will help to improve the performance of full-text queries. The performance of full-text queries is ...
Read more >
How to quickly search through a very large list of strings ...
Now, your problem is formulated as, having to search the text files which contains the set of strings. There are two possibilities here....
Read more >
Improving the performance of full-text search - Dropbox
For Firefly, Dropbox's full-text search engine, speed has always been a priority. (For more background on Firefly, check out our blog post).
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found