mopidy-youtube not able to read metadata from youtube.

Hello,

I have been using the youtube extension for a while; however, a month or more back it stopped doing searches and bringing back the metadata into existing playlists. I can create a valid Mopidy URI and add it and it will play from YT. And all existing URIs in playlists, still work but after a refresh of the playlists ( in IRIS) the video info is all gone and the URIs are shown as seen like this:

youtube:video:-NuX79Ud8zI
- * YouTube Video:

Since I had a workaround I just kept using it; however, with the update to mopidy I decided to dig in a bit on this issue. I even tried to pull the GIT feature/yt_playlist tree but it did not help, and seemed to have no effect.

I run mopidy as a service on a pi4b arm64 rpios.

Everything is updated, (I will put all versions and system info at bottom). I do not use the API.

Error returned from trying to update track info

Below shows the error returned from systemctl when looking up metadata. EG refresh a playlist that has a YT URI in it. I believe it puts out one error block for every YT URI it tries to update.

sudo systemctl status mopidy
---snip---
Jul 09 22:09:21 elan4 mopidy[5486]: INFO     [ThreadPoolExecutor-32_0] mopidy_youtube session.get triggered: list_videos (experimental)
Jul 09 22:09:21 elan4 mopidy[5486]: INFO     [ThreadPoolExecutor-32_0] mopidy_youtube session.get triggered: bs4api run_search
Jul 09 22:09:23 elan4 mopidy[5486]: INFO     [ThreadPoolExecutor-32_0] mopidy_youtube nothing in the soup, trying japi
Jul 09 22:09:23 elan4 mopidy[5486]: ERROR    [ThreadPoolExecutor-32_0] mopidy_youtube list_videos error "Extra data: line 1 column 84575 (char 84574)"
Jul 09 22:09:23 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube youtube LibraryProvider.lookup "youtube:video:-NuX79Ud8zI"
Jul 09 22:09:23 elan4 mopidy[5486]: [youtube] cBdNsC0BMT8: Downloading player 51ff6aac

Error Returned from using search

When using the search feature from the front end only searching Youtube. Nothing is returned. (eg search for big iron seen below). The page will show a message saying “Searching Youtube XYZ” where XYZ is at least Tracks and Playlist and the rest is too fast to see.

I have tried many searches using only letters (no spaces, numbers or special chars) I have tried URL’s and Video ID as well. all return nothing with the same type of error seen below (I think the “column X (char X-1)” changes). The systemctl status shows after search:

Jul 09 22:09:23 elan4 mopidy[5486]: [youtube] cBdNsC0BMT8: Downloading player 51ff6aac
Jul 09 22:24:46 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube youtube LibraryProvider.search "{'artist': ['Big Iron']}"
Jul 09 22:24:46 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube youtube LibraryProvider.search "{'album': ['Big Iron']}"
Jul 09 22:24:46 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube youtube LibraryProvider.search "{'any': ['Big Iron']}"
Jul 09 22:24:46 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube Searching YouTube for query "Big Iron"
Jul 09 22:24:46 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube session.get triggered: search
Jul 09 22:24:46 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube session.get triggered: bs4api run_search
Jul 09 22:24:47 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube nothing in the soup, trying japi
Jul 09 22:24:47 elan4 mopidy[5486]: ERROR    [YouTubeBackend-3] mopidy_youtube search error "Extra data: line 1 column 259817 (char 259816)"
Jul 09 22:24:47 elan4 mopidy[5486]: WARNING  [Core-11] mopidy.core.library YouTubeBackend does not implement library.search() with "exact" support. Please upgrade it.

Search for the video ID.

Jul 09 22:24:47 elan4 mopidy[5486]: WARNING  [Core-11] mopidy.core.library YouTubeBackend does not implement library.search() with "exact" support. Please upgrade it.
Jul 09 22:36:03 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube youtube LibraryProvider.search "{'artist': ['-NuX79Ud8zI']}"
Jul 09 22:36:03 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube youtube LibraryProvider.search "{'album': ['-NuX79Ud8zI']}"
Jul 09 22:36:04 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube youtube LibraryProvider.search "{'any': ['-NuX79Ud8zI']}"
Jul 09 22:36:04 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube Searching YouTube for query "-NuX79Ud8zI"
Jul 09 22:36:04 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube session.get triggered: search
Jul 09 22:36:04 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube session.get triggered: bs4api run_search
Jul 09 22:36:04 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube nothing in the soup, trying japi
Jul 09 22:36:04 elan4 mopidy[5486]: ERROR    [YouTubeBackend-3] mopidy_youtube search error "Extra data: line 1 column 23126 (char 23125)"
Jul 09 22:36:04 elan4 mopidy[5486]: WARNING  [Core-11] mopidy.core.library YouTubeBackend does not implement library.search() with "exact" support. Please upgrade it.

Search for just the letter a
Jul 09 22:55:57 elan4 mopidy[5486]: WARNING  [Core-11] mopidy.core.library YouTubeBackend does not implement library.search() with "exact" support. Please upgrade it.
Jul 09 23:22:32 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube youtube LibraryProvider.search "{'artist': ['a']}"
Jul 09 23:22:32 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube youtube LibraryProvider.search "{'album': ['a']}"
Jul 09 23:22:32 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube youtube LibraryProvider.search "{'any': ['a']}"
Jul 09 23:22:32 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube Searching YouTube for query "a"
Jul 09 23:22:32 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube session.get triggered: search
Jul 09 23:22:32 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube session.get triggered: bs4api run_search
Jul 09 23:22:33 elan4 mopidy[5486]: INFO     [YouTubeBackend-3] mopidy_youtube nothing in the soup, trying japi
Jul 09 23:22:33 elan4 mopidy[5486]: ERROR    [YouTubeBackend-3] mopidy_youtube search error "Extra data: line 1 column 230792 (char 230791)"
Jul 09 23:22:33 elan4 mopidy[5486]: WARNING  [Core-11] mopidy.core.library YouTubeBackend does not implement library.search() with "exact" support. Please upgrade it.

Using feature/yt_playlist

I read some of the closed issues, where it was stated that using the GIT feature/yt_playlist tree may help with search issues. This seemed to rev ytmusicapi from .15 to .18, but had no effect (tests above shown after proc below but are the same as before and I do not use the API).

I am not quite sure if I did it right. To use the tree I had to use sudo as mopidy is running as a service.

$ sudo python3 -m pip install git+git://github.com/natumbri/mopidy-youtube.git@feature/yt-playlists
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting git+git://github.com/natumbri/mopidy-youtube.git@feature/yt-playlists
  Cloning git://github.com/natumbri/mopidy-youtube.git (to revision feature/yt-playlists) to /tmp/pip-req-build-e0q7e8re
  Running command git clone -q git://github.com/natumbri/mopidy-youtube.git /tmp/pip-req-build-e0q7e8re
  Running command git checkout -b feature/yt-playlists --track origin/feature/yt-playlists
  Switched to a new branch 'feature/yt-playlists'
  Branch 'feature/yt-playlists' set up to track remote branch 'feature/yt-playlists' from 'origin'.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done

---snip all requirements met and download of ytmusicapi---

Installing collected packages: ytmusicapi
  Attempting uninstall: ytmusicapi
    Found existing installation: ytmusicapi 0.15.1
    Uninstalling ytmusicapi-0.15.1:
      Successfully uninstalled ytmusicapi-0.15.1
Successfully installed ytmusicapi-0.18.0

System Versions and settings:

Running "/usr/bin/mopidy --config /usr/share/mopidy/conf.d:/etc/mopidy/mopidy.conf deps" as user mopidy
Executable: /usr/bin/mopidy
Platform: Linux-5.10.46-v8+-aarch64-with-debian-10.10
Python: CPython 3.7.3 from /usr/lib/python3.7
Mopidy: 3.2.0 from /usr/lib/python3/dist-packages
---snip---
Mopidy-YouTube: 3.3 from /usr/local/lib/python3.7/dist-packages
  youtube-dl: 2021.2.10 from /usr/local/lib/python3.7/dist-packages
  cachetools: 4.2.1 from /usr/local/lib/python3.7/dist-packages
  Mopidy: 3.2.0 from /usr/lib/python3/dist-packages
  Pykka: 2.0.3 from /usr/lib/python3/dist-packages
  ytmusicapi: 0.18.0 from /usr/local/lib/python3.7/dist-packages
    requests: 2.25.1 from /usr/local/lib/python3.7/dist-packages
      certifi: 2018.8.24 from /usr/lib/python3/dist-packages
      chardet: 3.0.4 from /usr/lib/python3/dist-packages
      idna: 2.6 from /usr/lib/python3/dist-packages
      urllib3: 1.24.1 from /usr/lib/python3/dist-packages
  setuptools: 54.2.0 from /usr/local/lib/python3.7/dist-packages
  beautifulsoup4: 4.7.1 from /usr/lib/python3/dist-packages
  requests: 2.25.1 from /usr/local/lib/python3.7/dist-packages
    certifi: 2018.8.24 from /usr/lib/python3/dist-packages
    chardet: 3.0.4 from /usr/lib/python3/dist-packages
    idna: 2.6 from /usr/lib/python3/dist-packages
    urllib3: 1.24.1 from /usr/lib/python3/dist-packages
---snip---
GStreamer: 1.14.4.0 from /usr/lib/python3/dist-packages/gi
  Detailed information: 
    Python wrapper: python-gi 3.30.4
    Relevant elements:
      Found:
        uridecodebin
        souphttpsrc
        appsrc
        alsasink
        osssink
        oss4sink
        pulsesink
        id3demux
        id3v2mux
        lamemp3enc
        mpegaudioparse
        mpg123audiodec
        vorbisdec
        vorbisenc
        vorbisparse
        oggdemux
        oggmux
        oggparse
        flacdec
        flacparse
        shout2send
      Not found:
        flump3dec
        mad

Thank you for any help and for creating a really useful extension.

Side note: my workaround was to search on youtube and copy the video ID out of the URL. Bandcamp extension has a javascript link that scrapes and creates a Mopidy URI from pages… Something like that for YT would be cool and probably easier because you could use the URL to get the Video ID directly.

[natumbri]

Thanks @kingosticks - I don’t reckon that’s the problem, but fixing that regex up is a good idea in any case. I’ll look into it. (It might be the problem, but I have a feeling it is something else.)

I’m going to simplify things shortly, by getting rid of the regex and bs4 based APIs and keeping only the official API, the JSON based one and YouTube Music. Hopefully fewer of these types of issues to sort out!

[kingosticks]

Could it be that this regex is greedy and is matching more than just the json structure you want, which then fails when you try and parse it? Try changing it to:

r"ytInitialData = ({.*?});",

Or add some debugging around there and see what you are actually matching.