Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

User model will not scale with embeded lists of references to followers

See original GitHub issue

What version of Node.js are you using?


What version of Yarn are you using?


What browser are you using?


What operating system are you using?

Fedora 34

Describe the Bug

The user model has links to all followers and following. If one famous person uses the network, and has 15 million followers, their user document will have 171 megabytes of follower links (a mongodb _id is 12 bytes). Fetching the user model will be slow.

Reference to scalability issue in source code:

Other embedded references in the User document will cause similar issues. Over time a user might contribute lots of posts, messages, and comments which all cause the User document to grow without bounds.

Expected Behavior

A user with millions of followers will have a small User document in mongodb.

Steps to reproduce

  1. Create a popular social network

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

DimiMikadzecommented, Nov 22, 2021

@face, sorry for the late response. To be honest, after building Orca, I realized using a relational database like Postgres would make more sense since almost all the data is related to each other. Unfortunately, I don’t have time to do this refactoring at the moment.

I agree that having dozens of million users will cause scaling issues, but honestly, I don’t think anyone is using Orca at that scale.

Anyway, if you are open to sending a PR improving the user model scalability, I’d be glad to review and merge it.

Thanks once again for the detailed issue!

facecommented, Nov 23, 2022

Hi @DimiMikadze sorry my late response now, I’m on vacation again and playing with this problem.

The solution I came up with was tri database (didn’t use orca so I can’t open a PR 😦. To summarize, this is what I did to have a solution that would scale to millions followers:

1. Use Postgres for auth and a source of truth for any data which must be 100% consistent.
2. Use Cassandra for all social data
3. Use Redis for caching

Postgres is one of the best sources of truth, but doesn’t scale or shard well. Cassandra is great at scaling, sharding, and, grouping and finding “related” data. I had never used cassandra before but it’s awesome. Although it’s not a relational database, it’s better at finding related data than a relational database. You can specify part of your combination key for a table to dictate how the data is sorted and written to disk. So when you search for related data, it is a pre-sorted column of data to read from the actual db files. You can’t do relational things like joins in queries, so you pre-build structures like a user’s feed in a table (which is ultimately what you want for performance and scale).

Read more comments on GitHub >

github_iconTop Results From Across the Web

The Flask Mega-Tutorial, Part VIII: Followers, Contacts And ...
We said we wanted to have all users to have "followed" and "followers" lists. Unfortunately, a relational database does not have a list...
Read more >
Users API - GitLab Docs
Get a list of a specified user's count of projects, groups, issues and merge requests. Administrators can query any user, but non-administrators can...
Read more >
Create follower API | Elasticsearch Guide [8.5] | Elastic
When this API returns, the follower index exists, and cross-cluster replication starts replicating ... Note that certain settings can not be overrode (e.g., ......
Read more >
Humility in novice leaders: links to servant leadership and ...
ABSTRACT. Across two studies, we investigated connections between leader humility, servant leadership, and follower satisfaction.
Read more >
Good, Bad, and Ugly Leadership Patterns: Implications for Followers ...
When respondents were leaders (one of the samples in Study 1), the five behaviors were rated on a 5-point scale (1 = I...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found