scan_iter is *MUCH slower* than keys

Version: What redis-py and what redis version is the issue happening on? 2.10.5

Platform: What platform / version? (For example Python 3.5.1 on Windows 7 / Ubuntu 15.10 / Azure) CentOS 6.6 python 2.7.14

Description: Description of your issue, stack traces from errors and code that reproduces the issue According to https://redis.io/commands/keys:

Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don’t use KEYS in your regular application code. If you’re looking for a way to find keys in a subset of your keyspace, consider using SCAN or sets.

I wrote the following code for a comprision between scan_iter and keys, the result was very surprising to me, scan_iter is MUCH slower than keys:

import redis
import time
redis_cache = redis.Redis(host='my_redis_host)

time_start = time.time()
for i in range(100):
    for key in redis_cache.scan_iter(match='MONITORABLE-*', count=100):
        pass
print time.time() - time_start

time_start = time.time()
for i in range(100):
    for key in redis_cache.keys('MONITORABLE-*'):
        pass
print time.time() - time_start

Result:

1.26971101761
0.0261030197144

After I changing the cound from 100 to 10, it was even slower:

11.1533768177
0.0263831615448

My redis DB is not very large (totally around 500 keys), did I do anything wrong? Could you please give me some guide on this, thank you in advance!

[andymccurdy]

@WayneYe The problem with KEYS is that it pulls all matched keys into memory. If you have a Redis server with a lot of keys, this could be problematic and the client could run out of memory. The SCAN commands are instead pull matched keys into memory one chunk at a time.

SCAN is a memory optimization, not a CPU one.

[bowuL]

@WayneYe The problem with KEYS is that it pulls all matched keys into memory. If you have a Redis server with a lot of keys, this could be problematic and the client could run out of memory. The SCAN commands are instead pull matched keys into memory one chunk at a time.

SCAN is a memory optimization, not a CPU one.

waaa… get a very good answer