performance hit with NWB + fancy indexing

These lines can lead to slow reads from the NWB extractor due to fancy indexing. Whenever I do something like this, I read the full channel column and then do a fast fancy index from the ndarray in memory. It is technically wasteful, but the use case for memory mapping is to pull out relatively small chunks.

https://github.com/SpikeInterface/spikeinterface/blob/29a64a2f5024cdd11c127159c58ab84facd23af5/spikeinterface/extractors/nwbextractors.py#L226-L238

I refactored this downstream as the following. Big improvement when working with a ChannelSliceRecording

        if isinstance(channel_indices, slice):
            traces = es.data[start_frame:end_frame, channel_indices]
        else:
            # channel_indices is np.ndarray
            channel_indices = np.asarray(channel_indices)
            if channel_indices.size > 1:
                traces = es.data[start_frame:end_frame][:, channel_indices]
            else:
                traces = es.data[start_frame:end_frame, channel_indices[0]]

[miketrumpis]

Excellent points. Certainly 100x memory waste is unacceptable, and probably slower. I will def keep using a subclass for the scenario with <<2x memory overhead and 4x-8x speedup. The not-very-helpful proposal that occurs to me here is a threshold policy (e.g. over/under 2x overhead) for automatically deciding between the two approaches.

[alejoe91]

closed for inactivity