question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Housekeeping the storage file

See original GitHub issue

Hello

for long-running processes that do a lot of updates, the storage file can grow very large because NeDB appends the new document state to the file after every update.

{"_id":"djNwjBIrpw6O86dj","animal":"Polycorn","wizardLevel":10}
{"_id":"djNwjBIrpw6O86dj","animal":"Polycorn","wizardLevel":13}
{"_id":"djNwjBIrpw6O86dj","animal":"Polycorn","wizardLevel":22}
{"_id":"djNwjBIrpw6O86dj","animal":"Polycorn","wizardLevel":100}
{"_id":"djNwjBIrpw6O86dj","animal":"Polycorn","wizardLevel":11}
...
{"_id":"djNwjBIrpw6O86dj","animal":"Polycorn","wizardLevel":50}

After initializing the database (i.e. after restarting the application), NeDB cleans up the storage file and only stores the latest states.

{"_id":"djNwjBIrpw6O86dj","animal":"Polycorn","wizardLevel":50}

It would be really nice if this functionality could be called by the application to clean up the file every now and then. Or set an interval so that NeDB would do this periodically either after a certain amount of updates or based on time.

Like

db.housekeep();

or

db.setHousekeepingInterval(60 * 1000); // housekeep every minute

Issue Analytics

  • State:closed
  • Created 10 years ago
  • Comments:7 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
louischatriotcommented, Jul 15, 2013

I see what you mean. I added two methods to the Persistence object used by the datastore:

  • db.persistence.setAutocompactionInterval(interval), interval in milliseconds
  • db.persistence.stopAutocompaction

They do exactly what you want and their names are pretty explicit 😃 They are still undocumented, as I’m still wondering whether to expose them directly as methods of the datastore object or not, but you can use them already.

0reactions
alexgorbatchevcommented, Aug 13, 2013

Great stuff! This should be documented!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Linux system housekeeping 101: Managing file storage
One of your many duties as a system administrator is the often daunting task of keeping your system's filesystems clear of clutter.
Read more >
Housekeeping - IBM
Run the following housekeeping tasks regularly: Identify processing failures for command and data files; Archive and/or remove input files.
Read more >
Housekeeping - MicroStream Reference Manual
It is comprised of mechanisms for cleaning up storage files, clearing unneeded cached data and recognizing deleted entities via garbage collection.
Read more >
Clean Up Files Fast: Organize Your Online Storage ... - PCMag
Organize your messy files and folders with this foolproof strategy. ... It works for cleaning up your online storage, as well as tidying...
Read more >
Housekeeping: Records and Files - ASCCC
Don't bother cleaning out files less than two years old. These are relatively current, and you have a high likelihood of needing to...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found