Youtube-DL no longer picking up on Private/Deleted/Unavailable/Blocked videos in playlists

Checklist

• I’m reporting a broken site support issue
• I’ve verified that I’m running youtube-dl version 2021.04.07
• I’ve checked that all provided URLs are alive and playable in a browser
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar bug reports including closed ones
• I’ve read bugs section in FAQ

Verbose log

youtube-dl -v -ij --flat-playlist "https://www.youtube.com/playlist?list=PLAPG5SCuN8e0QopIC79GNubPZKrUGbkat"

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-ij', '--flat-playlist', 'https://www.youtube.com/playlist?list=PLAPG5SCuN8e0QopIC79GNubPZKrUGbkat']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2021.04.07
[debug] Python version 3.8.3 (CPython) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg N-93885-gc61d16cf3b, ffprobe N-93885-gc61d16cf3b, phantomjs 2.1.1
[debug] Proxy map: {}
{"_type": "url", "ie_key": "Youtube", "id": "7wfudXAZZ8I", "url": "7wfudXAZZ8I", "title": "More Stoner Ramblings From A Rock", "description": null, "duration": 209.0, "view_count": null, "uploader": "Humptydumptytribe"}

Description

As some of you might have noticed, in the last couple of days, Youtube tweaked their web interface and introduced a sneaky new configuration: Now, when you view a playlist, all private, deleted, blocked and unavailable videos are hidden by default. If you want to view these hidden entries, you have to click on the three dots under the playlist name on the webpage, then select “Show Unavailable Videos.” Some days back, Youtube-DL could still list all these hidden playlist items. But now, it’s no longer picking up on these items. Here is an example of a playlist with 11 videos, 10 of which have been made private: https://www.youtube.com/playlist?list=PLAPG5SCuN8e0QopIC79GNubPZKrUGbkat

I got Youtube-DL to list brief information for items in the playlist as follows: youtube-dl -ij --flat-playlist “https://www.youtube.com/playlist?list=PLAPG5SCuN8e0QopIC79GNubPZKrUGbkat”

But unfortunately, Youtube-DL ONLY picks up on the one playlist-item item which is available for download.

PLEASE, PLEASE would you sort out this bug and make Youtube-DL pick up on all hidden playlist items again? There are those of us who want to keep a record of Video ID’s which have been deleted/made private/made unavailable, for various reasons. There are often videos which are unavailable due to them simply being blocked in certain countries, even though they are playable and downloadable. Even these are hidden.

(After-Thought: I have a strong suspicion that Youtube made this recent change in order to cover up how many videos they’ve been deleting in their censorship campaign.)

[coletdjnz]

The screenshot is from Firefox, it renders the json in an interactive viewer.

[shoxie007]

Is this problem ever going to be fixed? I’m getting frustrated waiting for new releases. It seems like such a simple problem to solve, yet several weeks later no changes.

LATER EDIT: After sleeping over it, then waking up and analyzing the code again, I realize that much of the code will have to be overhauled and re-written to adapt to this latest change in Youtube. Now, instead of requesting a webpage for the first 100 or so results in a playlist, Youtube-DL will have to request a JSON with some extra arguments in the request command. I see a number of places in the code where changes will have to be made.

I went through the Youtube-DL code myself, tried to rectify the problem, then compiled and ran my own builds of Youtube-DL so that hidden playlist-items would be picked up, but I’ve had no success, at least when using command “youtube-dl -j --flat-playlist …”

Can anyone help me out? @colethedj are you familiar with the code for Youtube-DL? Where would I need to make the change so that I could get “youtube-dl -j --flat-playlist …” to pick up on all missing playlist items? I need to find some place in the code, in one of files youtube.py, YoutubeDL.py, common.py, utils.py or options.py where I can inject some extra data-arguments:

• params=‘wgYCCAA%3D’ (show unavailable)
• browseId=‘VL<playlist id>’

before Youtube-DL makes the webpage-requests for the playlist webpage or JSON.

This is as far as I’ve gotten: I’ve figured out that --flat-playlist would invoke function process_ie_result in YoutubeDL.py, cos this is the ONLY place where we find any mention of “extract_flat” from options.py. But prior to this, a webpage or JSON would have been downloaded and its info read. I’m not sure where that would be in the code.