[Freesound] Downloaded Audios

Checklist

• I’m requesting a site-specific feature
• I’ve verified that I’m running yt-dlp version 2022.06.22.1 (update instructions) or later (specify commit)
• I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
• I’ve searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I’ve read the guidelines for opening an issue
• I’ve read about sharing account credentials and I’m willing to share it if required

Region

Southern California, USA (not telling the exact town I’m living in for privacy reasons unless required)

Example URLs

https://freesound.org/people/C418/downloaded_sounds/

Description

Putting the url gives an error. I would like to download all audios made by C418 used in the MINECRAFT video game.

Verbose log

Run ia configure --username="$IA_EMAIL" --***
10
Config saved to: /home/runner/.config/internetarchive/ia.ini
11
[generic] Falling back on generic information extractor.
12
ERROR: Unsupported URL: https://freesound.org/people/C418/downloaded_sounds/
13
Traceback (most recent call last):
14
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1427, in wrapper
15
    return func(self, *args, **kwargs)
16
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1497, in __extract_info
17
    ie_result = ie.extract(url)
18
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 647, in extract
19
    ie_result = self._real_extract(url)
20
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/yt_dlp/extractor/generic.py", line 4136, in _real_extract
21
    raise UnsupportedError(url)
22
yt_dlp.utils.UnsupportedError: Unsupported URL: https://freesound.org/people/C418/downloaded_sounds/
23
24
Traceback (most recent call last):
25
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/tubeup/__main__.py", line 103, in main
26
    for identifier, meta in tu.archive_urls(URLs, metadata,
27
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/tubeup/TubeUp.py", line 415, in archive_urls
28
    downloaded_file_basenames = self.get_resource_basenames(
29
  File "/opt/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/tubeup/TubeUp.py", line 178, in get_resource_basenames
30
    if info_dict.get('_type', 'video') == 'playlist':
31
AttributeError: 'NoneType' object has no attribute 'get'
32
33
An exception just occured, if you found this exception isn't related with any of your connection problem, please report this issue to https://github.com/bibanon/tubeup/issues

[dirkf]

Example: the Freesound playlist extractor:

from .playlistbase import PlaylistBaseIE

class FreesoundPlaylistIE(PlaylistBaseIE):
    _VALID_URL = r'https?://(?:www\.)?freesound\.org/people/(?P<person>[^/]+)/(?P<type>downloaded_sounds|packs)/(?P<id>[^/]*)'

    _ITEM_RE = r'''(?s)<div\b[^>]+\bclass\s*=\s*("|')sound_filename\1[^>]*>\s*<a\b[^>]+\bclass\s*=\s*("|')title\2[^>]+\bhref\s*=\s*("|')(?P<url>(?:(?!\3).)+)'''
    _NEXT_RE = r'''<a\b[^>]+\bhref\s*=\s*("|')(?P<url>(?:(?!\1).)+)[^>]+\btitle\s*=\s*["']Next\s+Page'''
    _ITEM_IE = 'Freesound'

And playlistbase.py (using olde worlde yt-dl conventions):

# coding: utf-8
from __future__ import unicode_literals

import itertools
import re

from .common import InfoExtractor

from ..compat import (
    compat_kwargs,
)
from ..utils import (
    urljoin,
)


class PlaylistBaseIE(InfoExtractor):

    @property
    def _ITEM_RE(self):
        """
           Regex to match items
        """
        raise NotImplementedError('This attribute must be implemented by subclasses')

    @property
    def _NEXT_RE(self):
        """
           Regex to match next page URL
        """
        raise NotImplementedError('This attribute must be implemented by subclasses')

    @property
    def _ITEM_IE(self):
        """
           IE_KEY for items
           Specialise as required
        """
        return None

    def _next_page(self, pl_id, html, this_url=None, page_num=None):
        """
           Return URL of next page to be downloaded, or None
        """
        next_page = self._search_regex(self._NEXT_RE, html, 'next page link', group='url', default=None)
        return next_page and urljoin(this_url, next_page)

    def _item_get(self, match_obj):
        """
            Return item matched in match_obj
        """
        return match_obj.group('url')

    def _entries(self, url, pl_id, webpage=None):
        """
           Generate items to be downloaded
        """
        page_url = url
        for page in itertools.count(1):
            if not webpage:
                kwargs = {} if page < 2 else {'note': 'Downloading page %d' % (page, )}
                webpage = self._download_webpage(
                    page_url, pl_id, fatal=False, **compat_kwargs(kwargs))
            if not webpage:
                break
            for m in re.finditer(self._ITEM_RE, webpage):
                yield m
            next_page = self._next_page(pl_id, webpage, page_url, page)
            if not next_page or next_page == page_url:
                break
            page_url, webpage = next_page, None

    def _playlist_page_extract(self, url, pl_id, webpage):
        """
            Return playlist metadata: specialise as required
        """
        return {}

    def _real_extract(self, url):
        pl_id = self._match_id(url)
        webpage = self._download_webpage(url, pl_id)

        result = self.playlist_from_matches(
            self._entries(url, pl_id, webpage),
            playlist_id=pl_id, getter=lambda m: urljoin(url, self._item_get(m)), ie=self._ITEM_IE)
        result.update(self._playlist_page_extract(url, pl_id, webpage))
        return result

[dirkf]

Was thinking more of the paginated list pattern, where one might loop: start with a webpage, or have to download it, yield from re.finditer(...), then find or calculate the next page, or quit.

Variations if the entry list is a fancier type, if you want a fancier getter than lambda m: m.group('url'), etc.