MongoDB Writing Many Subscriptions at Once – Performance Issue

Describe the bug When switching from the standard YoutubeDL-Material database to the recommended Mongo database I am having issues keeping an uninterrupted database connection for an unknown reason.

To Reproduce Steps to reproduce the behavior:

1. Set up ~30 new subscriptions with the -w argument to not overwrite files (purpose: to make YTDLM catalogue subscriptions in MongoDB with no redownloading content)
2. Go to Settings -> Database in YTDLM web-gui and change the MongoDB from the standard string in the textbox, to the IP address, port and the trailing modifier for the ip/port number.
3. Test connection to DB, it works no problem
4. Observe the MongoDB Compass GUI program that is supposed to reflect changes of the DB on my server
5. There are no changes made on MongoDB Compass, I am stuck at 6.2k “Documents” catalogued.
6. Also YTDLM is using 100% of it’s available resources. I limited the contianer to 10gb of RAM and 2 cores

Expected behavior I expect to see maximum CPU and RAM output with no DB logging of “redownloaded with -w” files. I’ll mention again that the -w argument makes it so no file is overwritten, so in theory it has been simply only writing database changes into MongoDB. This spontaneously happened, and I didn’t adjust any settings from the point where the DB was writing normally and where it will not write anything now.

Screenshots If applicable, add screenshots to help explain your problem.

Environment Installed version: v4.2 - done You are up to date.

Installation type: docker Docker tag: nightly Commit hash: 88cc8d0 Build date: 2021-10-01

Ideally you’d copy the info as presented on the “About” dialogue in YoutubeDL-Material. (for that, click on the three dots on the top right and then check “installation details”. On later versions of YoutubeDL- Material you will find pretty much all the crucial information here that we need in most cases!)

Additional context imgur link of YTDLM DB settings and MongoDB Compass https://i.imgur.com/WVKvfxa.png Error Log: disregard the last four lines with MongoDB errors, I was experimenting to see if I could fix it with trying different links, it doesn’t work.

Settings
Main
Downloader
Extra
Database
Advanced
Users
Logs
2022-03-28T16:02:28.218Z INFO: Config items set using ENV variables.
2022-03-28T16:02:28.678Z INFO: YoutubeDL-Material v4.2 started on PORT 17442
2022-03-28T16:46:04.849Z ERROR: ERROR: [youtube] 1WPuKIByCSY: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] QLYtAXqyKzs: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] NMAl5hjDYIU: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] w7NsVm_Y9yA: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] j-Wk5fIdOrc: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] LSkJ80OLemY: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] A8ISC1Zx01Y: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] mkZK5grN01s: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] y75zv7uxXQ0: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] nXyQWbE2RfI: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] 7nJTt5tdKYM: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] 49trqlg6bMw: Sign in to confirm your age. This video may be inappropriate for some users.
2022-03-28T16:50:57.372Z ERROR: ERROR: [youtube] K5ECJwsgcTg: Sign in to confirm your age. This video may be inappropriate for some users.
2022-03-28T16:55:31.314Z INFO: Config items set using ENV variables.
2022-03-28T16:55:32.560Z INFO: YoutubeDL-Material v4.2 started on PORT 17442
2022-03-28T17:38:21.368Z ERROR: ERROR: [youtube] 1WPuKIByCSY: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] QLYtAXqyKzs: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] NMAl5hjDYIU: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] w7NsVm_Y9yA: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] j-Wk5fIdOrc: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] LSkJ80OLemY: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] A8ISC1Zx01Y: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] mkZK5grN01s: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] y75zv7uxXQ0: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] nXyQWbE2RfI: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] 7nJTt5tdKYM: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] 49trqlg6bMw: Sign in to confirm your age. This video may be inappropriate for some users.
2022-03-28T17:43:07.565Z ERROR: ERROR: [youtube] K5ECJwsgcTg: Sign in to confirm your age. This video may be inappropriate for some users.
2022-03-28T17:46:02.720Z INFO: Config items set using ENV variables.
2022-03-28T17:46:03.127Z INFO: YoutubeDL-Material v4.2 started on PORT 17442
2022-03-28T18:29:52.252Z ERROR: ERROR: [youtube] 1WPuKIByCSY: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] QLYtAXqyKzs: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] NMAl5hjDYIU: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] w7NsVm_Y9yA: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] j-Wk5fIdOrc: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] LSkJ80OLemY: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] A8ISC1Zx01Y: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] mkZK5grN01s: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] y75zv7uxXQ0: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] nXyQWbE2RfI: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] 7nJTt5tdKYM: Sign in to confirm your age. This video may be inappropriate for some users.
ERROR: [youtube] 49trqlg6bMw: Sign in to confirm your age. This video may be inappropriate for some users.
2022-03-28T18:35:27.384Z INFO: Config items set using ENV variables.
2022-03-28T18:35:27.770Z INFO: YoutubeDL-Material v4.2 started on PORT 17442
2022-03-28T18:41:00.558Z ERROR: connection timed out
2022-03-28T18:41:00.656Z ERROR: Failed to connect to MongoDB. Verify your connection string is valid.
2022-03-28T18:41:36.498Z ERROR: connection timed out
2022-03-28T18:41:36.499Z ERROR: Failed to connect to MongoDB. Verify your connection string is valid.

[l-ucky]

subscriptions causing too much load

Yes, I think this may be still true, there was a lot of resource usage with ~30 subscriptions rewriting data (not video).

you’re talking about transitioning to the MongoDB

Yes, this problem happened right now, apart of the debugging process for pulling the latest nightly build. I also hav 6.2k existing entries in MongoDB already

Glad you’re in contact with Tzahi. Remember to sleep, it’s good for you. I’ll be on standby!

[l-ucky]

I’m wondering what type of education you need to look at the code you cited and understand it to identify an issue. Also, I repulled the nightly build for ytdlm and started mongoDB again. https://i.imgur.com/Dtlwf52.png (good connection) however I’m unable to transfer the local files ytdlm organized into mongoDB https://i.imgur.com/KukaN1e.png