Default efficient row iteration

It seems like iterating over a chunked array is inefficient at the moment, presumably because we’re repeatedly decompressing the chunks. For example, if I do

    for pos, row in enumerate(data_root.variants):
        print(row)
        if pos == 1000:
            break

it takes several minutes (data_root.variants is a large 2D chunked matrix) but if I do

    for pos, row in enumerate(chunk_iterator(data_root.variants)):
        print(row)
        if pos == 1000:
            break

it takes less than a second, where

def chunk_iterator(array):
    """ 
    Utility to iterate over the rows in the specified array efficiently
    by accessing one chunk at a time.
    """
    chunk_size = array.chunks[0]
    for j in range(array.shape[0]):
        if j % chunk_size == 0:
            chunk = array[j: j + chunk_size][:]
        yield chunk[j % chunk_size]

To me, it’s quite a surprising gotcha that zarr isn’t doing this chunkwise decompression, and I think it would be good to do it by default. There is a small extra memory overhead, but I think that’s probably OK, given the performance benefits.

Any thoughts?

[alimanfoo]

Hi @jeromekelleher, I like your suggestion. As @jakirkham says, the cache for decoded chunks would partly solve this, but as you say iteration is actually a simpler requirement where you only cache one chunk at a time while you iterate through it. PR welcome.

[alimanfoo]

I’d suggest adding a test_iter() method on the TestArray class in the test_core module. E.g., something like:

    test_iter(self):
        params = (
            ((1000,), (100,)),
            ((100, 100), (10, 10)),
            # any other combination of shape and chunks you'd like to test
        )
        for shape, chunks in params:
            z = self.create_array(shape=shape, chunks=chunks, dtype=int)
            a = np.arange(np.product(shape)).reshape(shape)
            z[:] = a
            for expect, actual in izip_longest(a, z):
                assert_array_equal(expect, actual)