Question: MYSQL-db consumes 100% CPU
See original GitHub issueDear community, I run etherpad-lite on my server and experience crashes due to MYSQL consuming 100% of the CPU. Does anybody have tips for troubleshooting or tips what the cause could be?
I am not sure if this is an adequate place to post this – so feel free to close this post, if it doesn’t belong here.
Crash is caused by:
- mysql-database spiking to 100% CPU usage and staying at 100% until reboot/crash
- etherpad is not responding, pad cannot be edited anymore
- eventually server crashes
- memory use is below 40%
Crash situation:
- crashed after 45mins of continuous writing once, after a reboot the pad still crashed again a few times
- on a different day a different pad crashed after 1.30h of continuous writing
- both times there were 2 authors and about 7 other people reading (though not in read-only mode)
I am not sure, what causes this sudden mysql-CPU-spikes. I can’t find any suspicious errors corresponding to the crash time in the syslogs, mysql/error.logs or via journalctl. But maybe I am not looking for the errors in the right places?
Only after the pad crashed there where some error-logs regarding charset issues, e.g.:
nodejs[976]: #033[33m[2020-07-01 15:03:04.777] [WARN] client - #033[39mError: Failed assertion: mismatched apply: 21930 / 21927 -- { errorId: '...
But I think that was caused by the mysql-shutdown/reboot. Nevertheless I checked the mysql-charset settings:
- in etherpad-lite/settings.json:
"dbType" : "mysql", "dbSettings" : { … "charset": "utf8mb4" },
- etherpad_db > charset: latin1
- table “store” inside etherpad_db > charset: utf8mb4
- column “key” inside table “store” inside etherpad_db > charset: utf8mb4
- column “value” inside table “store” inside etherpad_db > charset: utf8mb4
Non-Crash Info:
- other pads work well, e.g. smaller ones with 2 authors + 1 lurker, with 2h+ of continuous writing
- a 15min testing session didn’t show any problems, where about 15+ people were online at the same time writing and reading
- I can’t recreate the crash with the the load tester (9 authors). Even when adding additional long text the pads work well over time. I ran the loadtest for about 30-40mins. Maybe I should try it out longer?
Etherpad-Infos:
- Etherpad-Lite V. 1.8.0
- in production mode / deployed as a service (https://www.howtoforge.com/tutorial/ubuntu-etherpad-editor-server-installation/)
- installed plugins: ep_autoscrolldown (with a few custom changes in the plugin’s jquery) / ep_etherpad-lite / ep_mypads / ep_user_fontsize_version_2
Server-Infos:
- @ Digital Ocean
- 2 vCPUs
- 2GB Memory
- 25GB Disk
- Ubuntu 18
- failed2ban installed
Thanks so much for any help! I am happy to give further information. Unfortunately the pad is no longer available. It is quite unfortunate that I don’t know how to recreate the issue yet – but I thought I’d post this question anyways, in case anyone has spontaneous tips. Best regards and take care.
Issue Analytics
- State:
- Created 3 years ago
- Comments:5 (2 by maintainers)
More importantly update Etherpad. There is a known issue with mysql in older versions.
I have updated to Etherpad version 1.8.4. as you suggested. We’ve been testing the pad for a while now and under circumstances where it had failed before. The problem did not return so I can confirm that the update has successfully fixed the problem! Thanks again for your help.