[Feature Request] Sites offering alternative simultaneous media streams

• I’m reporting a feature request
• I’ve verified that I’m running yt-dlp version 2021.06.08
• I’ve searched the bugtracker for similar feature requests including closed ones

Description

(Spun off from #343)

A handful of sites sometimes offer multiple media streams of a given type that are meant to be played simultaneously or as alternatives to each other. The biggest one is probably Mediasite, which often offers a screencast stream (and presentation slides) in addition to the video stream showing the speaker. Other such cases are possible: in the past, GDCVault offered an alternative audio track containing the translation of the speaker’s talk, and a video stream containing the slides; #347 is an issue with a site offering both subtitled and dubbed versions of the same video.

This is not very common, but when it happens, it’s something of a pain to support. Currently, alternative Mediasite streams are offered as separate formats with a negative preference value, which means they are not downloaded by default, even though we can do it just fine. Because it is nowhere even mentioned that they are available for download, they can be hard to discover (see ytdl-org/youtube-dl#20611, ytdl-org/youtube-dl#23003). Some kind of general framework for handling such cases would be useful.

Here’s my design sketch: each format can declare a set of streams it contains by including a 'streams' key in its dict containing a non-empty list of stream identifiers (strings). If two formats declare the same stream identifier, they shall be considered as containing two different quality versions of the same content. If a format doesn’t have a 'streams' key, it will be synthesised based on the 'acodec' and 'vcodec' keys: the list will contain 'audio' unless 'acodec' is 'none' and it will contain 'video' unless 'vcodec' is 'none'.

The meaning of best would then be modified, and a couple of other selectors added:

• best: Picks the single best pre-multiplexed format that contains all streams;
• allbest: For each stream offered by the download, picks the best format containing it, and downloads them separately;
• mergeallbest: For each stream offered by the download, picks the best format containing it, and merges them all afterwards. If merging is not possible, nothing is selected.

The default value of the -f option would then become mergeallbest/allbest. Analogous selectors for the worst formats could be provided as well.

---

List of extractors that could potentially benefit (with example URLs if possible):

• Animelab offers both Japanese-dubbed and English-dubbed audio streams, and both hard-subbed and non-subbed video (#347)
  • https://www.animelab.com/player/wonder-egg-priority-episode-2/
• Mediasite offers slide and screencast video streams
  • http://mediasite.uib.no/Mediasite/Play/90bb363295d945d6b548c867d01181361d?catalog=a452b7df-9ae1-46b7-a3ba-aceeb285f3eb (requires --no-check-certificates for the moment)
• FranceTV offers audio description tracks
  • https://www.france.tv/france-2/les-invisibles/les-invisibles-saison-1/2764261-garenne.html (not downloadable by myself; may be geoblocked or no longer available)
• SVTPlay offers something they call ‘Clear Speech’
  • https://www.svtplay.se/video/17164286/bergman-revisited/bergman-revisited-ariel-avsnitt-5
• YouTube serves alternative audio for some videos and some multifeed videos, currently handled as playlists (note the alternative stream has its own ID):
  • https://youtu.be/rs1WF2SkjuY: normal and audio description audio streams
  • https://www.youtube.com/watch?v=LnlKwzc_TNA: audio streams in seven different languages
  • https://www.youtube.com/watch?v=mp0TW8vkCLg, https://www.youtube.com/watch?v=7jQ5wLZ6Pj4: multifeed videos (currently each feed is handled as a separate video, as each has separate ID; this one may be left as-is)
• ArteTV is known to serve the same media dubbed into different languages (mostly French and German). V1 API returns versions with burned-in subtitles in different language versions (#3086) and audio description streams.
  • https://www.arte.tv/fr/videos/104351-002-A/serviteur-du-peuple-1-23/ (v1 API offers burned-in subtitle variants)
  • https://www.arte.tv/de/videos/102958-001-A/in-therapie-staffel-2-1-35/ offers French and German language audio, with AD (Hörfilm) and hard-of-hearing (Hörgeschädigte) variants for both languages
  • https://github.com/yt-dlp/yt-dlp/pull/3302#issuecomment-1112897663 also mentions each stream comes with a different type of subtitle (forced/visual-only, HOH, translation), which also need to be distinguished
• TF1 serves audio in multiple language versions (#982)
• Zoom offers screen recording stream to download, in parallel with camera stream (#2523): no links available, though

[barkoder]

Related to this issue, YouTube now allows some videos to have 2 audio tracks such as a descriptive audio track for the blind. So yt-dlp defaults to picking 251-1 only.

Both 251-0 and 251-1 should get muxed with 251-1 selected as the default audio track when the muxed output video file gets played.

$ yt-dlp -v -F rs1WF2SkjuY
[debug] Command-line config: ['-v', '-F', 'rs1WF2SkjuY']
[debug] yt-dlp version 2022.02.04 [c1653e9] (py2exe)
[debug] exe versions: ffmpeg 4.4.1-essentials_build-www.gyan.dev (setts), ffprobe 4.4.1-essentials_build-www.gyan.dev
[debug] Optional libraries: mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: rs1WF2SkjuY
[youtube] rs1WF2SkjuY: Downloading webpage
[youtube] rs1WF2SkjuY: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, codec:vp9.2, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] Available formats for rs1WF2SkjuY:
ID    EXT   RESOLUTION FPS |   FILESIZE    TBR PROTO | VCODEC           VBR ACODEC      ABR     ASR MORE INFO
------------------------------------------------------------------------------------------------------------------------------------------------------
sb2   mhtml 48x27          |                   mhtml | images                                       storyboard
sb1   mhtml 80x45          |                   mhtml | images                                       storyboard
sb0   mhtml 160x90         |                   mhtml | images                                       storyboard
139-0 m4a   audio only     |    6.71MiB    48k https | audio only           mp4a.40.5   48k 22050Hz [en] English descriptive, low, m4a_dash
139-1 m4a   audio only     |    6.71MiB    48k https | audio only           mp4a.40.5   48k 22050Hz [en] English original (default), low, m4a_dash
249-0 webm  audio only     |    6.77MiB    49k https | audio only           opus        49k 48000Hz [en] English descriptive, low, webm_dash
250-0 webm  audio only     |    7.90MiB    57k https | audio only           opus        57k 48000Hz [en] English descriptive, low, webm_dash
249-1 webm  audio only     |    6.62MiB    48k https | audio only           opus        48k 48000Hz [en] English original (default), low, webm_dash
250-1 webm  audio only     |    7.67MiB    55k https | audio only           opus        55k 48000Hz [en] English original (default), low, webm_dash
140-0 m4a   audio only     |   17.80MiB   129k https | audio only           mp4a.40.2  129k 44100Hz [en] English descriptive, medium, m4a_dash
140-1 m4a   audio only     |   17.80MiB   129k https | audio only           mp4a.40.2  129k 44100Hz [en] English original (default), medium, m4a_dash
251-0 webm  audio only     |   14.32MiB   104k https | audio only           opus       104k 48000Hz [en] English descriptive, medium, webm_dash
251-1 webm  audio only     |   13.90MiB   101k https | audio only           opus       101k 48000Hz [en] English original (default), medium, webm_dash
17    3gp   176x144      6 |   10.72MiB    78k https | mp4v.20.3        78k mp4a.40.2    0k 22050Hz 144p
394   mp4   256x144     25 |    8.47MiB    61k https | av01.0.00M.08    61k video only              144p, mp4_dash
160   mp4   256x144     25 |    5.27MiB    38k https | avc1.4d400c      38k video only              144p, mp4_dash
278   webm  256x144     25 |    9.03MiB    65k https | vp9              65k video only              144p, webm_dash
395   mp4   426x240     25 |   12.92MiB    94k https | av01.0.00M.08    94k video only              240p, mp4_dash
133   mp4   426x240     25 |   11.12MiB    80k https | avc1.4d4015      80k video only              240p, mp4_dash
242   webm  426x240     25 |   12.91MiB    93k https | vp9              93k video only              240p, webm_dash
396   mp4   640x360     25 |   24.98MiB   181k https | av01.0.01M.08   181k video only              360p, mp4_dash
134   mp4   640x360     25 |   21.53MiB   156k https | avc1.4d401e     156k video only              360p, mp4_dash
18    mp4   640x360     25 |   66.35MiB   482k https | avc1.42001E     482k mp4a.40.2    0k 44100Hz 360p
243   webm  640x360     25 |   32.07MiB   233k https | vp9             233k video only              360p, webm_dash
397   mp4   854x480     25 |   45.60MiB   331k https | av01.0.04M.08   331k video only              480p, mp4_dash
135   mp4   854x480     25 |   36.59MiB   266k https | avc1.4d401e     266k video only              480p, mp4_dash
244   webm  854x480     25 |   48.72MiB   354k https | vp9             354k video only              480p, webm_dash
398   mp4   1280x720    25 |   94.03MiB   684k https | av01.0.05M.08   684k video only              720p, mp4_dash
136   mp4   1280x720    25 |   62.20MiB   452k https | avc1.4d401f     452k video only              720p, mp4_dash
22    mp4   1280x720    25 | ~247.92MiB  1761k https | avc1.64001F    1761k mp4a.40.2    0k 44100Hz 720p
247   webm  1280x720    25 |   91.82MiB   668k https | vp9             668k video only              720p, webm_dash
399   mp4   1920x1080   25 |  173.87MiB  1265k https | av01.0.08M.08  1265k video only              1080p, mp4_dash
137   mp4   1920x1080   25 |  247.94MiB  1804k https | avc1.640028    1804k video only              1080p, mp4_dash
248   webm  1920x1080   25 |  169.39MiB  1232k https | vp9            1232k video only              1080p, webm_dash
400   mp4   2560x1440   25 |  608.94MiB  4430k https | av01.0.12M.08  4430k video only              1440p, mp4_dash
271   webm  2560x1440   25 |  618.00MiB  4496k https | vp9            4496k video only              1440p, webm_dash
401   mp4   3840x2160   25 |    1.28GiB  9511k https | av01.0.12M.08  9511k video only              2160p, mp4_dash
313   webm  3840x2160   25 |    1.81GiB 13517k https | vp9           13517k video only              2160p, webm_dash

[pukkandan]

This is a great idea and shouldn’t be too difficult to implement.

There are however a few things that need addressing:

1. Current options should keep working as is. So, bestvideo should always select the one best video stream irrespective of it’s stream type
2. At the same time, the user should be able to select the best of a particular stream type (say bestvideo.presentation)
3. User should also have the option to select the best of each video stream (say bestvideo.all or allbestvideo)
4. Changing best to mean the multiplexed format with ALL streams could cause compat issues. Say there is a website that provides audio+video merged and we add story boards for it. Now if someone was using -f best, it will throw error since now there is no single format with all stream types. So we should keep best to refer to any format with 1 video + 1 audio
5. allbest/mergeallbest creates an ambiguity. Since best refers to the best multiplexed format, allbest intuitively refers to the best of each stream-type that has multiplexed formats

Due to these, I propose that we keep the video|audio(|image?) selector as-is, and add the stream selector on top of this. So the format selector will look like:

r'''(?x)
                        (?P<merge>merge)?
                        (?P<which>b|w|all|best|worst)
                        (?P<what>v|a|video|audio)?
                        (?P<containing>\*)?
                        (?:\.(?<stream>all|\w+))?
                        (?:\.(?<n>[1-9]\d*))?
'''

Here’s how it would address the above points

1. Current options should keep working as is. So, bestvideo should always select the one best video stream irrespective of it’s stream type
2. At the same time, the user should be able to select the best of a particular stream type (say bestvideo.presentation)

bestvideo -> best video of any stream type best*.presentation -> best format (audio/video/per-merged) of type presentation bestvideo.screencast -> best video-only format of type screencast

1. User should also have the option to select the best of each video stream (say bestvideo.all or allbestvideo)

bestvideo.all -> Best of each type of video-only streams, downloaded seperately mergebest*.all -> Best of each type of stream merged into 1 file

1. Changing best to mean the multiplexed format with ALL streams could cause compat issues. Say there is a website that provides audio+video merged and we add story boards for it. Now if someone was using -f best, it will throw error since now there is no single format with all stream types. So we should keep best to refer to any format with 1 video + 1 audio

We haven’t changed the meaning of best, but we also now don’t have any selector to select “pre-merged format with all stream types”. I will need to think about how this can be added, or if it is even needed

1. allbest/mergeallbest creates an ambiguity. Since best refers to the best multiplexed format, allbest intuitively refers to the best of each stream-type that has multiplexed formats

best.all -> best pre-merged format of each stream type best*.all -> Best of each type of stream, whether audio, video or pre-merged all.screencast -> All formats of type screencast Note that all before and after the . has different meanings