url_encoded_fmt_stream_map not in self.player_config_args

Seen this on a few videos. Here is one example:

https://www.youtube.com/watch?v=AzguO9C8XG4

Getting a KeyError in mixins.py:apply_descrambler that is called from here in main.py:

        for fmt in stream_maps:
             if not self.age_restricted and fmt in self.vid_info:
                mixins.apply_descrambler(self.vid_info, fmt)
            mixins.apply_descrambler(self.player_config_args, fmt)

Digging into the response, it looks like player_config_args only contains:

dict_keys(['csi_page_type', 'fexp', 'watermark', 'host_language', 'cver', 'gapi_hint_params', 'enablecsi', 'cr', 'player_response', 'vss_host', 'c', 'hl', 'fflags', 'innertube_api_key', 'enablejsapi', 'innertube_context_client_version', 'loaderUrl', 'innertube_api_version', 'title'])

I poked around a bit and found that the stream descriptions are still there, but are found like:

json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats']

The response seems to be similar to videos that work but I’m not too familiar with the YouTube API. See some of the results in this key below:

[{'itag': 137, 'mimeType': 'video/mp4; codecs="avc1.640028"', 'bitrate': 351777, 'width': 1920, 'height': 1080, 'initRange': {'start': '0', 'end': '742'}, 'indexRange': {'start': '743', 'end': '1386'}, 'lastModified': '1565378295769874', 'contentLength': '10541869', 'quality': 'hd1080', 'fps': 6, 'qualityLabel': '1080p', 'projectionType': 'RECTANGULAR', 'averageBitrate': 310817, 'approxDurationMs': '271333', 'cipher': 'url=https%3A%2F%2Fr5---sn-nx5e6nez.googlevideo.com%2Fvideoplayback%3Fexpire%3D1576229251%26ei%3DIwXzXc7SBoGskgaKxq3wBw%26ip%3D2601%253A1c2%253A1300%253Ae90%253A1990%253A39b4%253Ae6e0%253Aa505%26id%3Do-ADqAku9FrYHucmtlpsqOGL-JutoEmQ3pPWdz5ZwHz5Xi%26itag%3D137%26aitags%3D133%252C134%252C135%252C136%252C137%252C160%252C242%252C243%252C244%252C247%252C248%252C278%26source%3Dyoutube%26requiressl%3Dyes%26mm%3D31%252C29%26mn%3Dsn-nx5e6nez%252Csn-nx5s7n76%26ms%3Dau%252Crdu%26mv%3Dm%26mvi%3D4%26pl%3D26%26gcr%3Dus%26initcwndbps%3D1797500%26mime%3Dvideo%252Fmp4%26gir%3Dyes%26clen%3D10541869%26dur%3D271.333%26lmt%3D1565378295769874%26mt%3D1576207547%26fvip%3D5%26keepalive%3Dyes%26fexp%3D23842630%26c%3DWEB%26txp%3D2316222%26sparams%3Dexpire%252Cei%252Cip%252Cid%252Caitags%252Csource%252Crequiressl%252Cgcr%252Cmime%252Cgir%252Cclen%252Cdur%252Clmt%26lsparams%3Dmm%252Cmn%252Cms%252Cmv%252Cmvi%252Cpl%252Cinitcwndbps%26lsig%3DAHylml4wRAIgcMDWk0p0AcnXXi5_UypvnFydatO76ePHXhHz6WNl3PUCIDBBdKk8kgdSYIUt3Rp-3yY36PAwVbITPxujZzsgFNK9&s=CLBLgALgxI2wwRgIhALESALmISt_moS2ZkH4oFW7kPoxA0ACjKIt1eVZzjfCtAiEAzmztiB4Q6cMZQ4b3v-TibQyBje1VGMEMmzg7yP2JvkQ%3D&sp=sig'}, {'itag': 248, 'mimeType': 'video/webm; codecs="vp9"', 'bitrate': 231667, 'width': 1920, 'height': 1080, 'initRange': {'start': '0', 'end': '219'}, 'indexRange': {'start': '220', 'end': '1077'}, 'lastModified': '1565378294874885', 'contentLength': '7590878', 'quality': 'hd1080', 'fps': 6, 'qualityLabel': '1080p', 'projectionType': 'RECTANGULAR', 'averageBitrate': 223809, 'colorInfo': {'primaries': 'COLOR_PRIMARIES_BT709', 'transferCharacteristics': 'COLOR_TRANSFER_CHARACTERISTICS_BT709', 'matrixCoefficients': 'COLOR_MATRIX_COEFFICIENTS_BT709'}, 'approxDurationMs': '271333', 'cipher': 'url=https%3A%2F%2Fr5---sn-nx5e6nez.googlevideo.com%2Fvideoplayback%3Fexpire%3D1576229251%26ei%3DIwXzXc7SBoGskgaKxq3wBw%26ip%3D2601%253A1c2%253A1300%253Ae90%253A1990%253A39b4%253Ae6e0%253Aa505%26id%3Do-ADqAku9FrYHucmtlpsqOGL-JutoEmQ3pPWdz5ZwHz5Xi%26itag%3D248%26aitags%3D133%252C134%252C135%252C136%252C137%252C160%252C242%252C243%252C244%252C247%252C248%252C278%26source%3Dyoutube%26requiressl%3Dyes%26mm%3D31%252C29%26mn%3Dsn-nx5e6nez%252Csn-nx5s7n76%26ms%3Dau%252Crdu%26mv%3Dm%26mvi%3D4%26pl%3D26%26gcr%3Dus%26initcwndbps%3D1797500%26mime%3Dvideo%252Fwebm%26gir%3Dyes%26clen%3D7590878%26dur%3D271.333%26lmt%3D1565378294874885%26mt%3D1576207547%26fvip%3D5%26keepalive%3Dyes%26fexp%3D23842630%26c%3DWEB%26txp%3D2316222%26sparams%3Dexpire%252Cei%252Cip%252Cid%252Caitags%252Csource%252Crequiressl%252Cgcr%252Cmime%252Cgir%252Cclen%252Cdur%252Clmt%26lsparams%3Dmm%252Cmn%252Cms%252Cmv%252Cmvi%252Cpl%252Cinitcwndbps%26lsig%3DAHylml4wRAIgcMDWk0p0AcnXXi5_UypvnFydatO76ePHXhHz6WNl3PUCIDBBdKk8kgdSYIUt3Rp-3yY36PAwVbITPxujZzsgFNK9&s=6LGLgALgxI2wwRQIhAKPAAJg42kAxqNCop1U7K_lWbKL8XAIaA_zzJMkYhpcsAiA8DK0HnGNtWdZT9BkwY8LNVO43-LetpkipIEtP9UXn5w%3D%3D&sp=sig'}, {'itag': 136, 'mimeType': 'video/mp4; codecs="avc1.4d401f"', 'bitrate': 194950, 'width': 1280, 'height': 720, 'initRange': {'start': '0', 'end': '740'}, 'indexRange': {'start': '741', 'end': '1384'}, 'lastModified': '1565378295771249', 'contentLength': '5863672', 'quality': 'hd720', 'fps': 6, 'qualityLabel': '720p', 'projectionType': 'RECTANGULAR', 'averageBitrate': 172884, 'approxDurationMs': '271333', 'cipher': 'url=https%3A%2F%2Fr5---sn-nx5e6nez.googlevideo.com%2Fvideoplayback%3Fexpire%3D1576229251%26ei%3DIwXzXc7SBoGskgaKxq3wBw%26ip%3D2601%253A1c2%253A1300%253Ae90%253A1990%253A39b4%253Ae6e0%253Aa505%26id%3Do-ADqAku9FrYHucmtlpsqOGL-JutoEmQ3pPWdz5ZwHz5Xi%26itag%3D136%26aitags%3D133%252C134%252C135%252C136%252C137%252C160%252C242%252C243%252C244%252C247%252C248%252C278%26source%3Dyoutube%26requiressl%3Dyes%26mm%3D31%252C29%26mn%3Dsn-nx5e6nez%252Csn-nx5s7n76%26ms%3Dau%252Crdu%26mv%3Dm%26mvi%3D4%26pl%3D26%26gcr%3Dus%26initcwndbps%3D1797500%26mime%3Dvideo%252Fmp4%26gir%3Dyes%26clen%3D5863672%26dur%3D271.333%26lmt%3D1565378295771249%26mt%3D1576207547%26fvip%3D5%26keepalive%3Dyes%26fexp%3D23842630%26c%3DWEB%26txp%3D2316222%26sparams%3Dexpire%252Cei%252Cip%252Cid%252Caitags%252Csource%252Crequiressl%252Cgcr%252Cmime%252Cgir%252Cclen%252Cdur%252Clmt%26lsparams%3Dmm%252Cmn%252Cms%252Cmv%252Cmvi%252Cpl%252Cinitcwndbps%26lsig%3DAHylml4wRAIgcMDWk0p0AcnXXi5_UypvnFydatO76ePHXhHz6WNl3PUCIDBBdKk8kgdSYIUt3Rp-3yY36PAwVbITPxujZzsgFNK9&s=FLBLgALgxI2wwRAIgX9NYAilrXhMqi2qoAsWV3uTY6YktgU8CG3IEij3HXVkCIHkap3oJtBVAD3J7LE_BcDhOOD95cMcHcIRcWtJXWdjF&sp=sig'}, ... }]

Specifically, here is one of the the (unquoted) URLs:

‘url=https://r5---sn-nx5e6nez.googlevideo.com/videoplayback?expire=1576229251&ei=IwXzXc7SBoGskgaKxq3wBw&ip=2601%3A1c2%3A1300%3Ae90%3A1990%3A39b4%3Ae6e0%3Aa505&id=o-ADqAku9FrYHucmtlpsqOGL-JutoEmQ3pPWdz5ZwHz5Xi&itag=137&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&source=youtube&requiressl=yes&mm=31%2C29&mn=sn-nx5e6nez%2Csn-nx5s7n76&ms=au%2Crdu&mv=m&mvi=4&pl=26&gcr=us&initcwndbps=1797500&mime=video%2Fmp4&gir=yes&clen=10541869&dur=271.333&lmt=1565378295769874&mt=1576207547&fvip=5&keepalive=yes&fexp=23842630&c=WEB&txp=2316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cgcr%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRAIgcMDWk0p0AcnXXi5_UypvnFydatO76ePHXhHz6WNl3PUCIDBBdKk8kgdSYIUt3Rp-3yY36PAwVbITPxujZzsgFNK9&s=CLBLgALgxI2wwRgIhALESALmISt_moS2ZkH4oFW7kPoxA0ACjKIt1eVZzjfCtAiEAzmztiB4Q6cMZQ4b3v-TibQyBje1VGMEMmzg7yP2JvkQ=&sp=sig’

I tried extracting the URL portion, grabbing the &s= argument value, computing the cipher, passing it to a stream, and attempting to download. It worked 1 time out 5. I seem to (occasionally?) get a 403: Forbidden error. It seems that sometimes the cipher field does not contain the &s= argument. I’m not sure what the proper way to handle that case is… Just pass the URL without appending a &sig argument?

I can share some code if you are interested but it is incredibly messy as I’m mostly working without understanding the big picture of how the API works.

Thanks for your hard work on this great library and hope this helps.

[vgoklani]

Hi @hradec I tried your patch, but the download doesn’t have any audio. Are you getting sound ?

[brommly]

You can put a try/except in apply_descrambler like this:

def apply_descrambler(stream_data, key):
    # ...
    try:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(',')
        ]
    except KeyError:
        stream_data[key] = json.loads(stream_data['player_response'])['streamingData']['adaptiveFormats']
    logger.debug(
        'applying descrambler\n%s',
        pprint.pformat(stream_data[key], indent=2),
    )

But, then it requires more cases in apply_signature to handle the different format. My code seems to work okay now, but it’s too much of a hack for me to make a pull request. I’m hoping someone can use this info and come up with a better solution.