[FEATURE] Proposed/Planned Maintenance Dashboard Features (Overview)

This “mega thread” is not for technical implementation, but rather the dashboard itself (UI/concepts to include, etc…) If you want to provide feedback about specific ideas, consider posting to the linked issue if it exists. If none exists yet, feel free to send me a short direct message, I’m not done creating and looking for existing tickets yet as I created this thread out of brainstorming. 😃 Thank you!

---

Is your feature request related to a problem? Please describe. This shall provide an overview of proposed and planned features for the new Maintenance Dashboard.

Its purpose should mostly be to check on automated maintenance tasks running in the background as well as providing an extensive way to manually trigger, control, check the status for and possibly otherwise interact with manual maintenance tasks. Also, to schedule them if desired.

Some tasks may run for one stage and then in a second step require the user for confirmation. e.g.: “found 3 database entries with missing video/audio data in the file system. Click more info to see a list” could let the user manually verify that deleting the entries is what they want. Conversely, a message prompting to verify deletion of hundreds of files MAY be the result of the user for example intentionally deleting a particular channel from their disk if they lost interest, however it might also be the result of their file system not responding to requests. e.g. their storage may be on a disk not accessible ATM. An automated deletion of entries would obviously be unwanted.

Feature candidates

• as described in the introduction above, a way to delete database entries for missing videos/audios on the disk
• #433
• I’ve noticed that sometimes the downloads lists has phantom entries. If the user is sure there should be nothing in it, maybe allow the downloads table to be reset. This obviously means the one-off downloads, not subscription downloads since you can never be quite sure if there isn’t some video being downloaded for a sub suddenly, however you’re very likely to correctly remember you didn’t issue anything recently that isn’t already downloaded.
• Manually trigger a youtube-dl binary update, respectively the same for any alternative youtube-dl, whatever the user is using
• Issue a maintenance mode, meaning finish all current one-off and subscription downloads / recordings of live videos gracefully (latter should probably also be something that is displayed somewhere if that’s going on, because live channels indicate it’s a download that could possibly go on for hours, severely delaying the possibility to enter a maintenance mode). Then don’t start new ones. This is to make sure that the db can be safely restarted or backed up since we don’t expect write operations to it or because we may want to measure the performance of ytdl-m with no downloads happening.
• Trigger replacing the thumbnail for file entries in the DB to be replaced from an online live source that may be deleted to a downloaded thumbnail. I’ve noticed that many older DB entries never had their thumbnail saved as using the local asset, rather still linking to the online thumbnail. This would fix older entries in the DB lacking the thumbnailPath as ticketed in this issue: #434
• Download old Twitch chats that may be missing if available. Same for YouTube Live videos? Maybe this ties into #287 a little bit.
• Download YouTube comments sections manually. Especially useful for downloads that happened before we had this feature eventually and also for manual uploads to the instance that weren’t sourced from the live page. Basically an extension of the support that has yet to be implemented as per issue #289
• Also allow for rebuild of thumbnails that have no file downloaded already. If the original thumbnail cannot be found at the URL its been downloaded from or if the video has been a local upload from disk allow for a thumbnail to be uploaded per video. Ties into #https://github.com/Tzahi12345/YoutubeDL-Material/issues/210#issuecomment-688687691
• Maybe the rebuild database command could be integrated in here as well.
• Run a backup job for the database (local and MongoDB should both be supported) and store to a /app/backups directory. Also, possibly have a restore option from here.
• Maybe also allow for the backup of all the archive files we have. (merged and the subscriptions archives)
• Manual check if any of our subs have missing videos and issue (re)downloads from there.
• Find subscribed channels with videos that are stored outside the subscription folder (e.g. manual uploads and previous one-off downloads) and then in another step allow for them to be merged into the subscriptions folder. Also think of extra files like thumbnails, subtitles, chat replays/comments, info.json, … (anything I’m missing? Probably 😛) — also refer to #423 for this A previous issue for this exists: #158 originally proposed to do this automatically, maybe offering to do this on-demand with a way to preview the actions would be better, especially since it allows the user to verify that all additional files like thumbnails and such moved as well.
• Manage video online checks at original source as per #288 (see this comment https://github.com/Tzahi12345/YoutubeDL-Material/issues/288#issuecomment-745338251 specifically)
• #139
• I once proposed to offer to store channel pages locally. (#428) as in: make snapshots of them as they appear on the platform and then pull from that stuff like channel art (banners, channel icon, graphics in descriptions like on Twitch, …) and About Me texts. It could be nice to set a schedule how often we check for updates to these, basically deciding how often we snapshot which channel and also to do this manually. (this item in the list basically means the management part of the scrape scheduling) Definitely have good rate limiting for this.

---

This list is a work in progress.

[Tzahi12345]

So I don’t think this can be done all at once, but we can prioritize specific tasks to be complete. I’m thinking for the first iteration:

1. Duplicates in DB fix, since it seems pressing
2. Add missing DB records for files
3. DB backup
4. Delete records with missing files
5. youtube-dl updates

Any thoughts?

[GlassedSilver]

It is by default! Or at least it should be

Ah cool, well I’m paranoid, so I’ve always left that, thumbnails and the other jazz in my custom flags. hah 😄