Computing numpy array from dask array taking very long time.

I am using MPII Human Pose Dataset having around 25k images of resolution 1280x720 in the form of dask array. So I’m converting it to a numpy array for further processing purposes. I’ve tried two approaches:

1. To process a single image it is taking around 3 min. Below is the code I’m using.

import hub ds = hub.load("username/MPII_Human_Pose_Dataset") ds['image'][1].compute()

1. Below approach takes 1 min for a single image.

import hub import numpy as np ds = hub.load("username/MPII_Human_Pose_Dataset") ds = pd.DataFrame.from_dict(ds) ds['image'][1].compute() or you can use np.asarray, it takes around 7 secs more than above approach. np.asarray(ds['image'][1])

Is there any way for reducing this much processing time? I only uploaded that dataset on the hub so can you check if the issue is from my side(in my #168 PR) or your side. (I’m using colab)

[AbhinavTuli]

Yup, as we discussed, chunk size of 1000 for high-resolution images was causing chunks of huge sizes to be created, causing the slowdown. Chunk size of 5 should fix this.

[sanchitvj]

Thanks for addressing the issue, everything is working fine now.