Differences in parsing with decode_responses=True, with and without hiredis

I ran into a similar problem as #809: I have a redis connection using decode_repsonses=True, but I am also using a 3rd-party package (RQ) that pickles some data and deep in that library I get UnicodeDecodeErrors. I’m kind of stuck because of that library’s use of pickled data.

However, I thought to install hiredis-py to see if it handled parsing differently and it does.

So, with this:

import redis
import pickle
rdb = redis.StrictRedis(decode_responses=True)
data = pickle.dumps(dict(hello='world'))
rdb.set('foo', data)
rdb.get('foo')

With hiredis installed I see:

b'\x80\x03}q\x00X\x05\x00\x00\x00helloq\x01X\x05\x00\x00\x00worldq\x02s.'

Without hiredis installed, I get:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

Looking at hiredis-py code, I see the equivalent of this:

if encoding:
   try:
      data = data.decode(encoding)
   except ValueError:
      pass

I’m wishing redis-py did the same. I’d love to see the following change to connection.py:

def decode(self, value, force=False):
   "Return a unicode string from the byte representation"
   if (self.decode_responses or force) and isinstance(value, bytes):
      try:
         value = value.decode(self.encoding, self.encoding_errors)
      except ValueError:
         # Ignore encoding and return bytes
         pass
   return value

Considering using hiredis comes highly recommended, I’d love to see this base difference in parsing resolved.

Thanks!

[piercefreeman]

@andymccurdy Stumbled upon this thread when investigating my own bug with decoding responses with binary data. Is the addition of a r.responses_as_bytes context manager still on the roadmap or have you switched approaches?

[andymccurdy]

The proposed change (redis/hiredis-py#82) to hiredis-py has been accepted and merged. Future versions of hiredis-py will behave sanely and no longer silently discard decoding errors by default.