Failure of export_to_phy from existing mountainsort cluster data

I have a use case where I take already data already sorted and curated with MountainSort, and funnel through SpikeInterface API to generate phy files…

import json, os, spikeinterface
import spikeinterface.core.waveform_extractor as wave
import spikeinterface.exporters.to_phy as phy
import spikeinterface.extractors.mdaextractors as mda

# Point to a prv recording and the params file
local_path = '/Volumes/GenuDrive/RY16_direct/MountainSort/RY16_36.mountain/RY16_36.nt1.mountain'
prv_file = os.path.join(local_path, 'raw.mda.prv')
params_file = os.path.join(local_path, 'params.json')

# Derive the properties in those json dicts
with open(prv_file, 'r') as F:
    raw_dict = json.load(F)
with open(params_file, 'r') as F:
    params_dict = json.load(F)

# Create the mda object
mda_reecording = mda.read_mda_recording(local_path,
                                   raw_fname=raw_dict['original_path'])
mda_reecording.annotate(is_filtered=True) # we've already filtered it, so mark it thus

# Derive the spikes file
firings_file = os.path.join(local_path, 'firings_raw.mda')
sorted_spikes = mda.read_mda_sorting(firings_file, sampling_frequency=params_dict['samplerate'])

# Create waveforms per spike
waveform_file = os.path.join(local_path, 'waveforms')
waveforms = wave.extract_waveforms(mda_reecording, sorted_spikes, waveform_file)

# Export to phy
phyplace = os.path.join(local_path, "phy")
phy.export_to_phy(waveforms, phyplace)

Which is great: I like phy a bit more than qt-mountainview curation. But for some of my data, this triggers an error in the export_phy step. Namely, recording chunk end points can overshoot the mda N1 x N2 size bounds, raising an error.

Disclaimer: I didn’t fully read the code in the execution stack to ensure this is kosher. But bounding the end to the true sample here fixes it. In other words, it works if I alter line 55 in job_tools.py …

def devide_recording_into_chunks(recording, chunk_size):
 51     all_chunks = []
 52     for segment_index in range(recording.get_num_segments()):
 53         num_frames = recording.get_num_samples(segment_index)
 54         chunks = divide_segment_into_chunks(num_frames, chunk_size)
 55         #all_chunks.extend([(segment_index, frame_start, frame_stop) for frame_start, frame_stop in chunks])
 57         # modification in the the next two lines 
 56         sample_max = recording.get_num_samples()
 57         all_chunks.extend([(segment_index, frame_start, np.min((frame_stop, sample_max)) for frame_start, frame_stop in chunks])
 58     return all_chunks

[samuelgarcia]

Did you try my working branch ?

pip uninstall spikeinterface
git clone https://github.com/samuelgarcia/spikeinterface.git
cd spikeinterface
git checkout some_fixes
python setup.py install

On my side there is no bug anymore.

[alejoe91]

I think this could be fixed by setting end_frame=num_samples if end_frame>num_samples in the get_traces function of the MdaRecordingSegment 😃