JSONDecodeError when trying to enter YouTube URL to wagtail admin

Found a bug? Please fill out the sections below. 👍

Issue Summary

Im having issues embedding the YouTube URL . It returns the error : JSONDecodeError Expecting value: line 1 column 1 (char 0) The version i am on is wagtail 2.9.3

Here is the full error trace back from my console:

  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\core\handlers\exception.py", line 34, in inner
    response = get_response(request)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\core\handlers\base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\core\handlers\base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\views\decorators\cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\admin\urls\__init__.py", line 109, in wrapper
    return view_func(request, *args, **kwargs)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\admin\auth.py", line 188, in decorated_view
    return view_func(request, *args, **kwargs)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\admin\views\pages.py", line 380, in edit
    if form.is_valid() and not page_perms.page_locked():
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\modelcluster\forms.py", line 315, in is_valid
    form_is_valid = super(ClusterForm, self).is_valid()
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\forms\forms.py", line 180, in is_valid
    return self.is_bound and not self.errors
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\forms\forms.py", line 175, in errors
    self.full_clean()
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\forms\forms.py", line 376, in full_clean
    self._clean_fields()
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\forms\forms.py", line 394, in _clean_fields
    value = field.clean(value)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\core\blocks\base.py", line 543, in clean
    return self.block.clean(value)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\core\blocks\stream_block.py", line 198, in clean
    (child.block.name, child.block.clean(child.value), child.id)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\core\blocks\stream_block.py", line 198, in clean
    (child.block.name, child.block.clean(child.value), child.id)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\core\blocks\struct_block.py", line 129, in clean
    result.append((name, self.child_blocks[name].clean(val)))
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\blocks.py", line 69, in clean
    if isinstance(value, EmbedValue) and not value.html:
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\django\utils\functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\blocks.py", line 22, in html
    return embed_to_frontend_html(self.url)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\format.py", line 9, in embed_to_frontend_html
    embed = embeds.get_embed(url)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\embeds.py", line 24, in get_embed
    embed_dict = finder(url, max_width)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\embeds.py", line 20, in finder
    return finder.find_embed(url, max_width=max_width)
  File "C:\Users\dream\Desktop\NoboCMS\backend\cms_env\lib\site-packages\wagtail\embeds\finders\oembed.py", line 64, in find_embed
    oembed = json.loads(r.read().decode('utf-8'))
  File "C:\Python38\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python38\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[04/Aug/2020 01:33:54] "POST /admin/pages/3/edit/ HTTP/1.1" 500 207887

Steps to Reproduce

My blocks.py

class VideoBlock(blocks.StreamBlock):
    '''rich text'''
    content = blocks.StructBlock(
            [
            ("title",blocks.CharBlock(required=True, help_text="Add your Title")),
            ("video", EmbedBlock())
            ]
    )

    class Meta:
        icon = "edit"
        label = "Video Block"

which is then used in my homepage model as

class HomePage(Page):
    '''home page model'''
    content = StreamField(
        [
            ("video",block.VideoBlock()),
        ] , blank=True
    )
    max_count = 1
    content_panels = [
        StreamFieldPanel('content'),
    ] + Page.content_panels 
    api_fields= [
        APIField('content'),
    ]

    class Meta:
        verbose_name = "Home Page"
        verbose_name_plural = "Home Pages"

To get the YouTube URL , i right clicked on the YouTube video url and clicked the copy video url option . The url is https://youtu.be/bbLlvoGbbEc . Upon clicking publish , the error mentioned above appeared

• I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: (yes / no) yes

Technical details

• Python version: 3.8.2
• Django version: 3.0.5
• Wagtail version: 2.9.3
• Browser version: Chrome 84

[gasman]

OK, now this makes sense! Wagtail is (perhaps wrongly…) using http instead of https for the endpoint that fetches the embed:

https://github.com/wagtail/wagtail/blob/master/wagtail/embeds/oembed_providers.py#L17

Normally this wouldn’t be a problem, because Youtube redirects the http URL to https, and Python’s urllib follows redirects automatically - but on your network the redirect isn’t happening, and it ends up trying to parse your home filter’s error message instead…

We should fix the URLs in oembed_providers.py to use https. In the meantime, you should be able to work around this by adding the following to your settings file:

from wagtail.embeds.oembed_providers import all_providers

youtube_fixed = {
    "endpoint": "https://www.youtube.com/oembed",
    "urls": [
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/watch.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/v/.+$',
        r'^http(?:s)?://youtu\.be/.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/user/.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/[^#?/]+#[^#?/]+/.+$',
        r'^http(?:s)?://m\.youtube\.com/index.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/profile.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/view_play_list.+$',
        r'^http(?:s)?://(?:[-\w]+\.)?youtube\.com/playlist.+$',
    ],
}

WAGTAILEMBEDS_FINDERS = [
    {
        'class': 'wagtail.embeds.finders.oembed',
        'providers': [youtube_fixed] + all_providers,
    }
]

[gasman]

I don’t think that’s related to the issue here. For support queries, please use the channels listed at https://docs.wagtail.io/en/stable/support.html.