Faster way to iterate all keys and values in db

I have a db with about 350,000 keys. Currently my code just loops through all keys and gets its value from the db.

However this takes almost 2 minutes to do, which seems really slow, redis-benchmark gave 100k reqs/3s.

I’ve looked at pipelining but I need each value returned so that I end up with a dict of key, value pairs.

At the moment I’m thinking of using threading in my code if possible to speed this up, is this the best way to handle this usecase?

Here’s the code I have so far.

import redis, timeit
start_time = timeit.default_timer()
count = redis.Redis(host='127.0.0.1', port=6379, db=9)
keys = api_count.keys()

data = {}

for key in keys:
    value = count.get(key)
    if value:
        data[key.decode('utf-8')] = int(value.decode('utf-8'))

elapsed = timeit.default_timer() - start_time

print('Time to read {} records: '.format(len(keys)), elapsed)

[andymccurdy]

Reducing the number of round trips to the server will certainly help. You can also take advantage of the SCAN command’s automatic pagination. Something like:

client = redis.StrictRedis(db=9, decode_responses=True)
data = {}
cursor = '0'
while cursor != 0:
    cursor, keys = client.scan(cursor=cursor)
    values = client.mget(*keys)
    values = map(int, values)
    data.update(dict(zip(keys, values)))

Alternatively, you could make this even easier/faster if you used a single redis hash (dictionary) to store all of these keys and values. Then use a single HGETALL command to retrieve the entire hash as a Python dictionary.

[jonathan-kosgei]

By adding count=1000000 to your code it completes in about 8 seconds for about 550k keys. Thanks!

  client = redis.StrictRedis(db=9, decode_responses=True)
  data = {}
  cursor = '0'
  while cursor != 0:
      cursor, keys = client.scan(cursor=cursor, count=1000000)
      values = client.mget(*keys)
      values = [value for value in values if not value == None]
      data.update(dict(zip(keys, values)))