question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

fast growing database since update 1.8.0 -> 1.8.7

See original GitHub issue

We run a large etherpad instance since many years. After updating from ep 1.8.0 to 1.8.7 the size of the database has nearly doubled the last 12 days. Usage was even less during holiday season (as seen in traffic monitoring).

3 days ago:

# ls -lh /var/lib/mysql/etherpadlite
total 44G
-rw-rw---- 1 mysql mysql   65 Feb 20  2013 db.opt
-rw-rw---- 1 mysql mysql 1.4K Jul 30  2018 store.frm
-rw-rw---- 1 mysql mysql  42G Jan  7 14:28 store.MYD
-rw-rw---- 1 mysql mysql 1.6G Jan  7 14:28 store.MYI

today:

# ls -lh /var/lib/mysql/etherpadlite
total 53G
-rw-rw---- 1 mysql mysql   65 Feb 20  2013 db.opt
-rw-rw---- 1 mysql mysql 1,4K Jul 30  2018 store.frm
-rw-rw---- 1 mysql mysql  51G Jän 10 18:34 store.MYD
-rw-rw---- 1 mysql mysql 1,7G Jän 10 18:34 store.MYI

File upload plugin is enabled, but that loads files into the files system (checked).

List of installed plugins

tables4
ep_image_upload
ep_toc
headings
slideshow
scrolltoanchor
code_formatting 
message_all
print
loading_message

Anybody observing similar grow of database size? Is there any ep tool to observe what is really going on in the database?

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:23 (22 by maintainers)

github_iconTop GitHub Comments

1reaction
webzwo0icommented, Mar 30, 2021

probably best to do it directly in the database. Not sure if recent mysql versions are better at json parsing (ideally we would only count the length of the pool value), but maybe the following is enough. It will (hopefully) select revs%100 with a timestamp matching 16104* which is between Jan 11 and Jan 13 2021. In case it does not work I would try to use something like 1610* or even 161*.

select store.key, length(store.value) from store 
where store.key like '%:revs:%00' and store.value like '%"timestamp":16104%'
order by length(store.value)+0 desc limit 100 ;

FYI: the largest value of all times on my instance is about 350kB. As written above, it includes the pad text, so very large pads impact the result. If you extract the results into a file you can extract the pool with jq '.meta.pool' filename and inspect it manually for very large values.

1reaction
JohnMcLearcommented, Jan 31, 2021

Yep this is about the behavior we expected from that change 😃 This matches our public test instance so all good here.

Thanks for reporting and sending graphs and updates etc 😃

Read more comments on GitHub >

github_iconTop Results From Across the Web

rasos - Profile - Bountysource
rasos commented on this issuefast growing database since update 1.8.0 -> 1.8.7. ether 1 year ago. rasos commented on this issueShare system audio....
Read more >
Apache Subversion 1.8 Release Notes
Apache server access log files will grow more quickly due to the larger number of requests and responses. As of 1.7.3, the httpd...
Read more >
Hsqldb User Guide
You can use Database Manager or other Java database access tools to explore your database and update it with SQL commands. These programs...
Read more >
Debezium connector for MySQL
After that intial snapshot is completed, the Debezium MySQL connector restarts from the same position in the binlog so it does not miss...
Read more >
Bug listing with status RESOLVED with resolution FIXED as at ...
... Bug:147 - "gdbm-1.8.0-r3.ebuild bug" status:RESOLVED resolution:FIXED severity: ... Bug:1244 - "moved the configuration data in /usr/sbin/etc-update" ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found