Improve (really) large libraries handling

Museeks is fine with small and medium libraries. However, there is always a OMGUbuntu! guy to come and say “I have a 6TB library, how would Museeks perform with it?”.

• Check any slowdown/bottlenecks:
  • During the scan
  • UI responsiveness in general
• Improve/fix what can be improved/fix

---

Things found so far:

UI/UX

• Better visual indicator when refreshing the library

On really large libraries (let’s say 100K files), you will not see a lot of progress in the footer. Maybe we could add a spinner, an animated progress bar or the count of scanned files.

• Libraries information/stats

In the settings view, Library tag, we could add more information about the library: the number of files, the total size of it…

---

Scan

• Check how we could boost up things even more

Other id3 libraries (#371), improve the “get duration” operation…

• Add tracks to the libraries during Library scan

Reports were made of some frustration with big libraries being scanned and no possibility to start playing music right away. Because it could be a costly render/thing, maybe add a bulk of 100/1000 tracks at a time.

---

UI navigation performances

• Improve scroll in TracksList https://github.com/KeitIG/museeks/blob/master/src/ui/components/Shared/TracksList.react.js#L162

Scroll in Trackslist is CPU consuming to keep some basic operations like track selection fast and in O(1). It forces us to split tracks into chunks and to render them. Maybe we could improve things a bit more here, by tracking what is being re-rendered and what is not.

• Library’s mapStateToProps https://github.com/KeitIG/museeks/blob/master/src/ui/components/Library/Library.react.js#L104

So we moved the search/sorting logic here because it is simpler to handle that way. There are inconsistencies with how other tracks list are used in the store (playlist are already sorted for example). At every Library.react re-render, these “costly” operations are made. Maybe by re-moving things to actions we can solve both perfs and KISS problems.

• ~[OPEN QUESTION] Improve DB performances: ditch IndexedDB/LinvoDB3 and switch to something else, like Dexie, PouchDB or sqlite?~ Linvo is fine

LinvoDB3 is not the fasted when it comes to full-database loading, I would be curious to see how other files-based DB compare.

[lachlan-00]

I just want to put a note on the issue for reference and as a thank you.

I now get a much better response even over sshfs. Museeks will scan my folder then start adding into the client as soon as it’s scanned the folder.

Overall this is a much better experience for large libraries and is usable as soon as 100 tracks are imported. Performance over sshfs is probably on par with rhythmbox so it’s a lot faster now than before.

[martpie]

I am closing the issue for now, I won’t implement the “Library information/stats” now as it may bring database schemes update and the feature does not seems that needed.

Feel free to discuss it if you think the opposite, and I’ll re-open it (or will open a separate issue)