question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Sessionstorage is constantly growing

See original GitHub issue

The session storage seems to be constantly growing When running etherpad we see a constantly growing number of session storage values in the database.

Is there a way to clean them up? We already looked into the script, which didn’t help. https://github.com/ether/etherpad-lite/blob/develop/src/bin/deleteAllGroupSessions.js . So it seems that this sessions are no group sessions.

A standard session storage entry looks like:

| sessionstorage:----KdezcUojWiGfuzBgo1L-OnF4SbbZ | {"cookie":{"path":"/","_expires":null,"originalMaxAge":null,"httpOnly":true,"sameSite":"Lax","secure":false}} |
| sessionstorage:---0sPTO0IIvtWBpbYpccywxSImVqSf_ | {"cookie":{"path":"/","_expires":null,"originalMaxAge":null,"httpOnly":true,"sameSite":"Lax","secure":false}} |
| sessionstorage:---2pUJfjIE00yXpIkrzvHvUEOCLKI2X | {"cookie":{"path":"/","_expires":null,"originalMaxAge":null,"httpOnly":true,"sameSite":"Lax","secure":false}} |

Server (please complete the following information):

  • Etherpad version: 1.8.13
  • OS: Debian Buster
  • Node.js version (node --version): v14.16.1
  • npm version (npm --version): 6.14.12

Additional context

All session storage keys values from mariadb:

MariaDB [etherpad]> SELECT COUNT(*) FROM store WHERE `key` LIKE 'sessionstorage%';
+----------+
| COUNT(*) |
+----------+
| 16306299 |
+----------+
1 row in set (14.974 sec)

None session storage keys values from mariadb:

MariaDB [etherpad]> SELECT COUNT(*) FROM store WHERE `key` NOT LIKE 'sessionstorage%';
+----------+
| COUNT(*) |
+----------+
| 54156416 |
+----------+
1 row in set (36.645 sec)

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:6 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
xshadowcommented, Apr 17, 2021

Thanks for etherpad as software and the fast reply. Thanks for pointing me to this issue.

If I calculated this correct, those 16 million session won’t require more space than 300MB, so this is fine. But I think it would be great if they could expire automatically or by or a session garbage collector job 😃

0reactions
rhansencommented, Dec 10, 2021

Design idea:

  • express-session config changes:
    • set maxAge, value TBD (maybe 7 days so the user doesn’t have to keep logging in?)
    • set rolling=true so that a session stays alive as long as the user stays connected
    • set saveUninitialized=false to lower the load on the db (though it might not help in our case because I think we always modify the session object)
    • set resave=false to lower the load on the db (though it might not help due to rolling=true)
  • SessionStore changes:
    • implement the touch() method
    • when creating/updating/touching a session, (re)set a timer that auto-deletes the session’s db record when it expires
  • call req.session.touch() each time the server receives a socket.io ping or message from the client

A problem with the above scheme: The constant .touch() calls might increase the load on the db considerably. We could reduce db load by skipping a db write unless it would extend the saved lifetime by more than some threshold. (In other words: Trade accuracy for reduced load.) For example, we could set maxAge to 14 days and only update the db record if it would extend the lifetime by 7 days or more. That should reduce db load to one write per week per session, which is trivial.

Another problem: We would need to clean up old session records after a dirty shutdown, or after upgrading from a version of Etherpad that doesn’t expire sessions. ueberdb doesn’t have cursor support so we can’t just iterate over all records that match sessionstorage:*. We could add cursor support to ueberdb, but that would take a lot of effort. Alternatively, with some clever key prefixing we can iterate over old sessions in small batches. Here’s one approach:

  • Add two new numeric db records:
    • instanceId: A counter that is incremented each time Etherpad starts up.
    • caughtUpToInstanceId: Used to clean up session state from previous Etherpad runs. Always less than or equal to instanceId.
  • When saving a session, use key `sessionstore:${instanceId}:${sid}` (instead of the current `sessionstorage:${sid}`) and delete records `sessionstore:${caughtUpToInstanceId}:${sid}` through `sessionstore:${instanceId - 1}:${sid}`.
  • When looking up a session ID, fetch all keys `sessionstore:${caughtUpToInstanceId}:${sid}` through `sessionstore:${instanceId}:${sid}` (inclusive) in parallel. Of the non-null results, use the one with the highest instance ID.
  • On startup, launch a worker thread to clean up or migrate session state from previous runs:
    for (; caughtUpToInstanceId < instanceId; ++caughtUpToInstanceId) {
      // The following query should return a small number of keys (the number of
      // sessions that were active when instance caughtUpToInstanceId exited).
      for (const key of getAllDbKeysWithPrefix(`sessionstore:${caughtUpToInstanceId}:`)) {
        const sess = getDbValue(key);
        deleteDbRecord(key);
        // If the session is still valid, save under the current instanceId and start a cleanup timer.
        if (!isSessionExpired(sess)) saveSession(sess);
      }
    }
    

To clean up legacy `sessionstorage:${sid}` records, we could do something like this:

const alphabet = 'abcdefghijklmnopqrstuvwxyz';
const sidCharset = `_-0123456789${alphabet}${alphabet.toUpperCase()}`;
for (const chars of cartesianProductGenerator(Array(4).fill(sidCharset))) {
  for (const key of getAllDbKeysWithPrefix(`sessionstorage:${chars.join('')}`) {
    deleteDbRecord(key);
  }
}      

But it would probably be better to just have the user issue a native DB query to delete all of the legacy records.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Removing 1Item from a Growing Array stored in Session Storage
So I have an array stored in session storage that is constantly increasing, like adding items to a shopping cart, in this case...
Read more >
Session Storage is Evil - Liferay
Session storage is not evil from a developers perspective, but they are absolutely evil from deployment and runtime perspectives. Here's a short list...
Read more >
Local Storage vs Session Storage vs Cookie - XenonStack
Each browser storage method has a specific maximum data size. Both storage provide a large memory capacity. To be more specific, local Storage ......
Read more >
Figuring out sessionStorage size limit of a browser - SitePoint
I was able to figure out the size of sessionStorage that is used by my app using the following line of code in...
Read more >
Local Storage vs. Session Storage vs. Cookies
Session storage can store data ranging between 5mb - 10mb. The exact amount of storage capacity varies as per each browser's implementation, but...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found