[Broken]Facebook private (friends only and private groups) error handling response is broken

Checklist

• I’m reporting a broken site support
• I’ve verified that I’m running youtube-dlc version 2020.10.26
• I’ve checked that all provided URLs are alive and playable in a browser
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar issues including closed ones

Verbose log

./testdlc -v -F https://www.facebook.com/100002659934141/videos/3355692847862680/
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-F', u'https://www.facebook.com/100002659934141/videos/3355692847862680/']
[debug] Loading archive file None
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dlc version 2020.10.25
[debug] Python version 2.7.16 (CPython) - Darwin-19.6.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[facebook] 3355692847862680: Downloading webpage
[facebook] 3355692847862680: Downloading webpage
[facebook] 3355692847862680: Downloading webpage
ERROR: Cannot parse data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dlc -U  to update. Be sure to call youtube-dlc with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "./testdlc/youtube_dlc/YoutubeDL.py", line 830, in extract_info
    ie_result = ie.extract(url)
  File "./testdlc/youtube_dlc/extractor/common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "./testdlc/youtube_dlc/extractor/facebook.py", line 484, in _real_extract
    video_id, fatal_if_no_video=True)
  File "./testdlc/youtube_dlc/extractor/facebook.py", line 380, in _extract_from_url
    raise ExtractorError('Cannot parse data')
ExtractorError: Cannot parse data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dlc -U  to update. Be sure to call youtube-dlc with the --verbose flag and include its complete output.

Description

Private videos used to work fine until lately. I assume the server response from FB changed or something. As a result, ytdlc does not how to handle it and ask for cookies or login, for example.

[ssaqua]

Hey @someziggyman and FYI @blackjack4494

I was able to resolve this issue by changing one regex. I was considering making a pull request but I don’t really know how this change affects the rest of the extractor and other types of videos on FB.

Here’s a test link, in case needed for private group video: https://www.facebook.com/1051184515/videos/10220691460290341/

Could you test if this change works for your case?

Find this section in extractor/facebook.py#L366 and replace the regex for the fb_dtsg param. Should also supply credentials or use a cookie header of course.

            # Video info not in first request, do a secondary request using
            # tahoe player specific URL
            tahoe_data = self._download_webpage(
                self._VIDEO_PAGE_TAHOE_TEMPLATE % video_id, video_id,
                data=urlencode_postdata({
                    '__a': 1,
                    '__pc': self._search_regex(
                        r'pkg_cohort["\']\s*:\s*["\'](.+?)["\']', webpage,
                        'pkg cohort', default='PHASED:DEFAULT'),
                    '__rev': self._search_regex(
                        r'client_revision["\']\s*:\s*(\d+),', webpage,
                        'client revision', default='3944515'),
                    'fb_dtsg': self._search_regex(
-                       r'"DTSGInitialData"\s*,\s*\[\]\s*,\s*{\s*"token"\s*:\s*"([^"]+)"',
+                       r'"MRequestConfig"\s*,\s*\[\]\s*,\s*{\s*"dtsg"\s*:\s*{\s*"token"\s*:\s*"([^"]+)"',
                        webpage, 'dtsg token', default=''),
                }),
                headers={
                    'Content-Type': 'application/x-www-form-urlencoded',
                })

[vctls]

Than tried this Workaround ytdl-org/youtube-dl#27062 (comment) “youtube-dl --force-generic-extractor [link] and worked just fine.”

@rgime on what endpoints does that work? What do your URLs look like? Is it something like that? https://www.facebook.com/<user_ID>/videos/<video_ID> Or like this? https://www.facebook.com/groups/<group_IP>/permalink/<post_ID>/

Is it really private? Does it ask for authentication at all? What version are you using?