Getting 500 Internal Server error on some MTV videos

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

• I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

• I’m reporting a bug unrelated to a specific site
• I’ve verified that I’m running yt-dlp version 2022.09.01 (update instructions) or later (specify commit)
• I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I’ve read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

Some MTV episodes are giving a 500 internal Server Error, but others work.

Working: https://www.mtv.com/episodes/vg52l9/teen-mom-young-pregnant-new-beginnings-season-3-ep-24 Not Working: https://www.mtv.com/episodes/ignte9/teen-mom-the-next-chapter-wishful-thinking-season-1-ep-1 Not working: https://www.mtv.com/episodes/g230me/teen-mom-the-next-chapter-you-go-do-you-season-1-ep-2

Provide verbose output that clearly demonstrates the problem

• Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
• Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[root@MYSERVER Import]# yt-dlp -vU --cookies-from-browser Firefox "https://www.mtv.com/episodes/g230me/teen-mom-the-next-chapter-you-go-do-you-season-1-ep-2"
[debug] Command-line config: ['-vU', '--cookies-from-browser', 'Firefox', 'https://www.mtv.com/episodes/g230me/teen-mom-the-next-chapter-you-go-do-you-season-1-ep-2']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.09.01 [5d7c7d6] (zip)
[debug] Python 3.8.12 (CPython 64bit) - Linux-4.18.0-305.el8.x86_64-x86_64-with-glibc2.2.5 (glibc 2.28)
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: sqlite3-2.6.0
[Cookies] Extracting cookies from firefox
[debug] Extracting cookies from: "/root/.mozilla/firefox/p42buod4.default-default/cookies.sqlite"
[Cookies] Extracted 131 cookies from firefox
[debug] Proxy map: {}
[debug] Loaded 1670 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.09.01, Current version: 2022.09.01
yt-dlp is up to date (2022.09.01)
[debug] [mtv] Extracting URL: https://www.mtv.com/episodes/g230me/teen-mom-the-next-chapter-you-go-do-you-season-1-ep-2
[mtv] teen-mom-the-next-chapter-you-go-do-you-season-1-ep-2: Downloading webpage
[mtv] 0f294ac5-27d4-11ed-ad8e-0e40cf2fc285: Downloading info
ERROR: [mtv] g230me: Unable to download XML: HTTP Error 500: Internal Server Error (caused by <HTTPError 500: 'Internal Server Error'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 670, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/mtv.py", line 327, in _real_extract
    videos_info = self._get_videos_info(mgid, url=url)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/mtv.py", line 231, in _get_videos_info
    return self._get_videos_info_from_url(info_url, video_id, use_hls)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/mtv.py", line 234, in _get_videos_info_from_url
    idoc = self._download_xml(
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 1030, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 994, in download_handle
    res = self._download_webpage_handle(
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 864, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 821, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 803, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3656, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib64/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib64/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error

[bashonly]

For devs:

At least for the US version of the site, the _FEED_URL being used by the MTV extractor no longer works (http://www.mtv.com/feeds/mrss/) - this is what causes the 500 error

Instead, the webpage’s JSON now holds an API URL template, with a base of: https://topaz.viacomcbs.digital/topaz/api/mgid:

The API URL template can be found in the JSON like this:

main_container = self._extract_child_with_type(data, 'MainContainer')
flex_wrapper = self._extract_child_with_type(main_container, 'FlexWrapper') 
player = self._extract_child_with_type(flex_wrapper, 'Player')
api_url_template = traverse_obj(player, ('props', 'videoDetail', 'videoServiceUrl'))

The query template can be stripped from the URL and a query dict can be passed to _download_json instead. The XHR sent by the webpage adds these query params to the API URL: ssus, clientPlatform, browser, device, os

ssus can be a random UUID (use random_uuidv4()) clientPlatform seems to always be mobile browser can be Unknown device can be UNKNOWN os can be Unknown

I have not been able to test the other MTV sites due to geo-blocking

[devz3ro]

This is now happening on all MTV videos.

Confirmed an issue on all MTV videos. Anyone have an idea of a work around?