[Broken] Nebula

Checklist

• I’m reporting a broken site support
• I’ve verified that I’m running yt-dlp version 2021.05.20
• I’ve checked that all provided URLs are alive and playable in a browser
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar issues including closed ones

Verbose log

[debug] Command-line config: ['--verbose', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://nebula.app/videos/joe-scott-mysteries-strangest-epidemics-of-all-time']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.05.20 (exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.19042-SP0
[debug] exe versions: ffmpeg 4.3.2-2021-02-20-essentials_build-www.gyan.dev, ffprobe 4.3.2-2021-02-20-essentials_build-www.gyan.dev
[debug] Proxy map: {}
[debug] [Nebula] Extracting URL: https://nebula.app/videos/joe-scott-mysteries-strangest-epidemics-of-all-time
[Nebula] Logging in
[Nebula] joe-scott-mysteries-strangest-epidemics-of-all-time: Authenticating to Nebula with supplied credentials
[Nebula] joe-scott-mysteries-strangest-epidemics-of-all-time: Downloading webpage
[Nebula] joe-scott-mysteries-strangest-epidemics-of-all-time: Retrieving Zype API key
[Nebula] joe-scott-mysteries-strangest-epidemics-of-all-time: Retrieving metadata from Zype
[Nebula] joe-scott-mysteries-strangest-epidemics-of-all-time: Retrieving Zype access token
[debug] [Zype] Extracting URL: https://player.zype.com/embed/607da7862508000001b3322e.html?access_token=7ff86f9aed13fb2dca5cf5aac7fadaaaebc2a81d85eb8c77e5dd1d08b1e61610
[Zype] 607da7862508000001b3322e: Downloading JSON metadata
[Zype] 607da7862508000001b3322e: Downloading m3u8 information
ERROR: An extractor error has occurred. (caused by KeyError('URI')); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "yt_dlp\extractor\common.py", line 564, in extract
  File "yt_dlp\extractor\zype.py", line 100, in _real_extract
  File "yt_dlp\extractor\common.py", line 1896, in _extract_m3u8_formats
  File "yt_dlp\extractor\common.py", line 1922, in _extract_m3u8_formats_and_subtitles
  File "yt_dlp\extractor\common.py", line 2113, in _parse_m3u8_formats_and_subtitles
  File "yt_dlp\extractor\common.py", line 2046, in extract_media
KeyError: 'URI'
Traceback (most recent call last):
  File "yt_dlp\extractor\common.py", line 564, in extract
  File "yt_dlp\extractor\zype.py", line 100, in _real_extract
  File "yt_dlp\extractor\common.py", line 1896, in _extract_m3u8_formats
  File "yt_dlp\extractor\common.py", line 1922, in _extract_m3u8_formats_and_subtitles
  File "yt_dlp\extractor\common.py", line 2113, in _parse_m3u8_formats_and_subtitles
  File "yt_dlp\extractor\common.py", line 2046, in extract_media
KeyError: 'URI'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1126, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1147, in __extract_info
  File "yt_dlp\extractor\common.py", line 583, in extract
yt_dlp.utils.ExtractorError: An extractor error has occurred. (caused by KeyError('URI')); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.

Description

Whenever I try downloading content from Nebula, I get this KeyError. Beyond that, I’m not too sure what’s going wrong specifically. I’ve tried both watchnebula.com and nebula.app in the URL, since it recently changed, but the results are identical. I’m pretty sure you need an account to watch nebula content, and although I think you might be able to watch content with a free account, if you need to test through my paid account, please PM me, my Discord and Twitter are on my profile.

[pukkandan]

ok, gotcha. In that case, its the issue of zype not obeying the specs 😠 and we can just ignore the subs. You solution will work fine for it. Just so that this doesn’t happen again in the future, it would be beneficial to do a similar check for language. Something like lang = media.get('LANGUAGE') or 'unknown' should work.

Do you want to make a PR, or should I just cherry pick ur commit? Edit: I pushed the patch

[hheimbuerger]

@pukkandan @shawnshyguy First of all, I can reproduce this. It affects not only the video Shawn mentions, but also the existing test cases.

As you’ve probably already spotted on the posted stacktrace, the issue here lies somewhat in the Zype extractor that the Nebula extractor is based upon. @pukkandan, maybe you want to re-tag this issue as Zype?

The Zype extractor tries to extract subtitles from the m3u8: https://github.com/yt-dlp/yt-dlp/blob/c8633c89823545342df872385592e0326a95a217/yt_dlp/extractor/common.py#L2043-L2050

However, the subtitles on this Nebula video have an empty URI attribute:

#EXTM3U
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",URI="",LANGUAGE="en"
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=360448,RESOLUTION=426x240,CODECS="avc1.42001e,mp4a.40.2",SUBTITLES="subs"
https://mf.zype.com/[...]

According to https://datatracker.ietf.org/doc/html/rfc8216#section-4.3.4.2.1, I believe this isn’t valid:

The URI attribute of the EXT-X-MEDIA tag is REQUIRED if the media type is SUBTITLES

I’ve made a very quick attempt (i.e. a blame on a single line) to trace the base code back, and it seems to have been added 25 days ago in yt_dlp, by @fstirlitz, here: https://github.com/yt-dlp/yt-dlp/commit/a0c3b2d5cf4fe374a2b0119ea53b71c9a06aaee9 Again, this code is correct according to spec, just not quite matching reality. 😉

Therefore, my current (unproven) working theory is that the issue has always existed in Nebula/Zype, but only now has been revealed due to this addition.

Which I guess is why the code isn’t trying to handle this case more gracefully so far. We could just skip it, here’s an example implementation: https://github.com/yt-dlp/yt-dlp/compare/master...hheimbuerger:issue339-fix-nebula-extractor

This works for me, but I know next to nothing about these manifest formats, how the subtitle support works, etc. So I have no idea what the wider consequences of this change are, and if I deal with this exception in the right way.

Does that help you to resolve it, @pukkandan? Would you like me to open a PR for my commit? Or should I attempt to take another route and/or run more tests for you?