Huge long term disk usage (performance improvement)

Hello, on Linux, there is more than 100MB/s disk usage returned by iotop utility for the process: searchd --config /home/me/rats-search/sphinx.conf --nodetach searchd = rats-search/imports/linux/x64/searchd when i have killed the rats (there is no restart/shutdown button) and started again, i see after some minutes issue is back. Any idea on commands that can shed some light on this please? here some more details (password: r) i am at 1.7 million torrents and the database folder is 7,5G already, program shows something under 20 million torrents possible

[sanikolaev]

Hi guys. Manticore team member here.

Regarding “maybe now they can resolve something or give advice”: as it turned out the main problem may be that string attributes are used instead of stored fields. E.g. here https://github.com/DEgITx/rats-search/blob/master/src/background/sphinx.js#L44 and below that line all these are string attributes:

  index torrents
    rt_attr_string = hash
    rt_attr_string = name
    rt_attr_string = ipv4
    rt_attr_string = contentType
    rt_attr_string = contentCategory

  index files
      rt_attr_string = path
    rt_attr_string = hash
    rt_attr_string = size

and:

• they have to be in memory for good performance
• they take much more memory than numbers

Here and here I don’t find any signs that you sort or group by these attributes. I may be wrong, but if I’m not it may be beneficial for you app’s users if you store the above strings in “stored only fields” - http://mnt.cr/stored_only_fields

[DEgITx]

@slrslr , this issue is closed because it was reopened only for resolving/optimize some suggestions from manticore teams memebers, itself it’s not garantee that you problem is resolved on such big database, and I said before the problem related to manticore engine and not for rats search (so thats why this issue closed).

You need to ask @sanikolaev and and other manticore teams is any more optimization in you case possible to make such big database work with satisfactory speed and performance, because descresing of performance related to manticore. If they will tell that there is no possible optimization in configuration in database and it structure your only choise will be delete some data from tables to make you work comfatable. It’s possible to do with filter tab in rats search.