MongoDB server crashes several times a week

Crashes were much more frequent when mongodb+nodejs were running on a 2gb-ram instance, than now on a 3gb-ram instance. But still, I need to manually run sudo service mongodb restart several times a week, whenever openwhyd users see a “502 Bad Gateway” page while trying to access openwhyd.org. (e.g. https://www.facebook.com/groups/openwhyd/permalink/2056648404616708/)

Hypothesis:

1. whydjs (our nodejs app) sometimes takes too much memory (e.g. when running data joints in memory) too quickly, so Mongodb just panics
2. the system is not configurated adequately for mongodb
3. mongodb could run entirely from our SSD drive --> no need for RAM?
4. could upgrading to the latest version of mongodb solve this issue?

Next steps:

• download and read /var/log/mongodb/mongod.log
• apply mongodb’s “warning” recommendations (Disable Transparent Huge Pages (THP)) then reboot droplet
• find solutions on how to reduce mongodb memory usage (to try if it crashes again)
• wait 1 week to check if applying mongodb’s recommendations prevented crashes, or try other solutions
• if above solutions don’t fix the issue, check out mongodb’s changelog and consider upgrading
• update openwhyd’s server maintainance cookbook, to reflect changes

[coty-crg]

Just letting you know, you’re not alone. I have my scripts just auto-restart mongo at this point; it randomly crashes after running for long periods of time! I can only assume it’s due to the small amount of memory available to it, but that’s not an acceptable reason.

[adrienjoly]

The server has not crashed for 2 weeks! 🎉