User model will not scale with embeded lists of references to followers

What version of Node.js are you using?

v14.18.1

What version of Yarn are you using?

1.22.10

What browser are you using?

Firefox

What operating system are you using?

Fedora 34

Describe the Bug

The user model has links to all followers and following. If one famous person uses the network, and has 15 million followers, their user document will have 171 megabytes of follower links (a mongodb _id is 12 bytes). Fetching the user model will be slow.

Reference to scalability issue in source code: https://github.com/ElevenSymbols/orca/blob/536dc81096fcd2799eb3ed599c392769296b3540/packages/orca-api/src/db/follow.ts#L17

Other embedded references in the User document will cause similar issues. Over time a user might contribute lots of posts, messages, and comments which all cause the User document to grow without bounds.

Expected Behavior

A user with millions of followers will have a small User document in mongodb.

Steps to reproduce

1. Create a popular social network

[DimiMikadze]

@face, sorry for the late response. To be honest, after building Orca, I realized using a relational database like Postgres would make more sense since almost all the data is related to each other. Unfortunately, I don’t have time to do this refactoring at the moment.

I agree that having dozens of million users will cause scaling issues, but honestly, I don’t think anyone is using Orca at that scale.

Anyway, if you are open to sending a PR improving the user model scalability, I’d be glad to review and merge it.

Thanks once again for the detailed issue!

[face]

Hi @DimiMikadze sorry my late response now, I’m on vacation again and playing with this problem.

The solution I came up with was tri database (didn’t use orca so I can’t open a PR 😦. To summarize, this is what I did to have a solution that would scale to millions followers:

1. Use Postgres for auth and a source of truth for any data which must be 100% consistent.
2. Use Cassandra for all social data
3. Use Redis for caching

Postgres is one of the best sources of truth, but doesn’t scale or shard well. Cassandra is great at scaling, sharding, and, grouping and finding “related” data. I had never used cassandra before but it’s awesome. Although it’s not a relational database, it’s better at finding related data than a relational database. You can specify part of your combination key for a table to dictate how the data is sorted and written to disk. So when you search for related data, it is a pre-sorted column of data to read from the actual db files. You can’t do relational things like joins in queries, so you pre-build structures like a user’s feed in a table (which is ultimately what you want for performance and scale).