User model will not scale with embeded lists of references to followers
See original GitHub issueWhat version of Node.js are you using?
v14.18.1
What version of Yarn are you using?
1.22.10
What browser are you using?
Firefox
What operating system are you using?
Fedora 34
Describe the Bug
The user model has links to all followers and following. If one famous person uses the network, and has 15 million followers, their user document will have 171 megabytes of follower links (a mongodb _id
is 12 bytes). Fetching the user model will be slow.
Reference to scalability issue in source code: https://github.com/ElevenSymbols/orca/blob/536dc81096fcd2799eb3ed599c392769296b3540/packages/orca-api/src/db/follow.ts#L17
Other embedded references in the User document will cause similar issues. Over time a user might contribute lots of posts, messages, and comments which all cause the User document to grow without bounds.
Expected Behavior
A user with millions of followers will have a small User document in mongodb.
Steps to reproduce
- Create a popular social network
Issue Analytics
- State:
- Created 2 years ago
- Comments:6 (2 by maintainers)
Top Results From Across the Web
The Flask Mega-Tutorial, Part VIII: Followers, Contacts And ...
We said we wanted to have all users to have "followed" and "followers" lists. Unfortunately, a relational database does not have a list...
Read more >Users API - GitLab Docs
Get a list of a specified user's count of projects, groups, issues and merge requests. Administrators can query any user, but non-administrators can...
Read more >Create follower API | Elasticsearch Guide [8.5] | Elastic
When this API returns, the follower index exists, and cross-cluster replication starts replicating ... Note that certain settings can not be overrode (e.g., ......
Read more >Humility in novice leaders: links to servant leadership and ...
ABSTRACT. Across two studies, we investigated connections between leader humility, servant leadership, and follower satisfaction.
Read more >Good, Bad, and Ugly Leadership Patterns: Implications for Followers ...
When respondents were leaders (one of the samples in Study 1), the five behaviors were rated on a 5-point scale (1 = I...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@face, sorry for the late response. To be honest, after building Orca, I realized using a relational database like Postgres would make more sense since almost all the data is related to each other. Unfortunately, I don’t have time to do this refactoring at the moment.
I agree that having dozens of million users will cause scaling issues, but honestly, I don’t think anyone is using Orca at that scale.
Anyway, if you are open to sending a PR improving the user model scalability, I’d be glad to review and merge it.
Thanks once again for the detailed issue!
Hi @DimiMikadze sorry my late response now, I’m on vacation again and playing with this problem.
The solution I came up with was tri database (didn’t use orca so I can’t open a PR 😦. To summarize, this is what I did to have a solution that would scale to millions followers:
Postgres is one of the best sources of truth, but doesn’t scale or shard well. Cassandra is great at scaling, sharding, and, grouping and finding “related” data. I had never used cassandra before but it’s awesome. Although it’s not a relational database, it’s better at finding related data than a relational database. You can specify part of your combination key for a table to dictate how the data is sorted and written to disk. So when you search for related data, it is a pre-sorted column of data to read from the actual db files. You can’t do relational things like joins in queries, so you pre-build structures like a user’s feed in a table (which is ultimately what you want for performance and scale).