Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

UnicodeDecodeError while fetching items

See original GitHub issue

It seems like I randomly get errors like this:

 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xde in position 174: invalid continuation byte

        at msgpack._cmsgpack.Unpacker._unpack (_unpacker.pyx:443)
        at msgpack._cmsgpack.Unpacker.__next__ (_unpacker.pyx:518)
        at mpdecode (/usr/local/lib/python3.7/site-packages/scrapinghub/hubstorage/serialization.py:33)
        at iter (/usr/local/lib/python3.7/site-packages/scrapinghub/client/proxy.py:115)

This happens while iterating the items through last_job.items.iter() It seems to happen about 50% of the time from what I see. I scrape the same website every day and run that function and sometimes it works fine, sometimes raise that error. I am not sure if this is an issue with this library or with the ScrapingHub API though but it is very problematic.

This happens on the latest (2.3.1) version

Issue Analytics

State:
Created 3 years ago
Comments:8 (3 by maintainers)

Top GitHub Comments

1reaction

mijamocommented, Mar 1, 2021

It does seem like it fixed the issue, thank you for the help.

This might be worth mentioning in the documentation somewhere though because the error doesn’t make it easy to understand the problem.

1reaction

mijamocommented, Feb 18, 2021

Thank you I will try that solution and close this issue if it fixes it. It might take a few days though as the error doesn’t happen every day as I mentioned.