Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Read One Large Numpy Array

See original GitHub issue

Hey there, how would you recommend reading one large numpy array that does not fit into memory? I was thinking of just running:

arr=np.load(x,mmap_mode='r')
# pseudo code below
new_arr=da.concatenate([da.concatenate([arr[chunk_x,chunk_y] for chunk_x in x_chunks]) for chunk_y in y_chunks])

The chunk parts are just intervals that I’ve omitted and replaced with pseudocode to illustrate my point. Could this be done? Is there a feature to directly import numpy array? I’m not sure how the numpy stacking helps, especially if I want to use one file??

Issue Analytics

State:
Created 4 years ago
Comments:14 (7 by maintainers)

Top GitHub Comments

1reaction

TomAugspurgercommented, May 30, 2019

Can you give a full example?

1reaction

mrocklincommented, May 23, 2019

It sounds like you might just want da.from_array

On Thu, May 23, 2019 at 12:03 PM Tom Augspurger notifications@github.com wrote:

Can you rechunk the memmapped array with da.rechunk? I’m not sure what performance / memory usage will be like in that case.

dask_arr = da.from_array(arr).rechunk(…)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dask/dask/issues/4837?email_source=notifications&email_token=AACKZTGIXRQDF4JMKWFVSXDPW3E5BA5CNFSM4HPIITM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWC3OXI#issuecomment-495302493, or mute the thread https://github.com/notifications/unsubscribe-auth/AACKZTAA5CHCY3LGGK3MBUDPW3E5BANCNFSM4HPIITMQ .