Memoryviews and compression

Currently there are some issues with how we handle memoryviews.

1. We assume that len(mv) == mv.nbytes in several places. This is not the case for non-trivial shape or itemsize
2. We slice into memoryviews in at least one place (see distributed/protocol/utils.py), which is also not correct for non-trivial shape or itemsize
3. To support these concerns the Numpy serialization code currently always produces memoryviews that have strides (1,). This loses important information that stops intelligent compression.

I think that ideally we would propagate itemsize and strides information on memoryviews until after we pass through compression. Then we might consider flattening memoryviews before they enter the network layer (tornado.iostream.IOStream.write) or perhaps earlier. https://stackoverflow.com/questions/44486048/how-to-flatten-a-memoryview

Currently things seem safe but inefficient when itemsize might be useful.

[mrocklin]

This was helpful. Thank you. It looks like this problem only arises when we are moving enough data around.

On Wed, Jun 14, 2017 at 3:42 AM, Simon Perkins [e-mail] wrote:

Note that there are some arrays with zero dimensions in them, but commenting them out did not prevent the problem occurring.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/1159#issuecomment-308347825, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszGPDibfurpBLqDaZA7ZcMq-dHWYmks5sD47mgaJpZM4N2c4H .

[mrocklin]

We now pass through dtype information. We don’t yet pass through strides. I’ll wait on strides until we have a concrete application that needs N-d compression. Closing