Kilosort error: sgemm in CUBLAS failed

Hello Spikeinterface team, thank you in advance for all your support!

So I’m getting this error when running Kilosort3: (Matlab R2018a, CUDA 9.0, Visual Studio 2015, NVIDIA RTX A5000)

KilosortBase._run_from_folder <class 'spikeinterface.sorters.kilosort3.kilosort3.Kilosort3Sorter'>
RUNNING SHELL SCRIPT: _output_KS3\run_kilosort3.bat
Traceback (most recent call last):
  File "C:\Users\huynh\.conda\envs\si_env\lib\site-packages\spikeinterface\sorters\basesorter.py", line 200, in run_from_folder
    SorterClass._run_from_folder(output_folder, sorter_params, verbose)
  File "C:\Users\huynh\.conda\envs\si_env\lib\site-packages\spikeinterface\sorters\kilosortbase.py", line 42, in _run_from_folder
    raise Exception(f'{cls.sorter_name} returned a non-zero exit code')
Exception: kilosort3 returned a non-zero exit code

---------------------------------------------------------------------------
SpikeSortingError                         Traceback (most recent call last)
Input In [16], in <cell line: 2>()
      1 start_time = time.time()
----> 2 sorting_KS3 = ss.run_sorter('kilosort3', recording_saved, output_folder= '_output_KS3')
      3 print("--- _output_KS3 took %s seconds ---" % (time.time() - start_time))
      4 print(f'Kilosort3 found {len(sorting_KS3.get_unit_ids())} units')

File ~\.conda\envs\si_env\lib\site-packages\spikeinterface\sorters\runsorter.py:67, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, with_output, **sorter_params)
     61 if singularity_image is not None:
     62     return run_sorter_container(sorter_name, recording, 'singularity', singularity_image,
     63                                 output_folder=output_folder,
     64                                 remove_existing_folder=remove_existing_folder,
     65                                 delete_output_folder=delete_output_folder, verbose=verbose,
     66                                 raise_error=raise_error, with_output=with_output, **sorter_params)
---> 67 return run_sorter_local(sorter_name, recording, output_folder=output_folder,
     68                         remove_existing_folder=remove_existing_folder,
     69                         delete_output_folder=delete_output_folder,
     70                         verbose=verbose, raise_error=raise_error, with_output=with_output,
     71                         **sorter_params)

File ~\.conda\envs\si_env\lib\site-packages\spikeinterface\sorters\runsorter.py:89, in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params)
     86 SorterClass.set_params_to_folder(
     87     recording, output_folder, sorter_params, verbose)
     88 SorterClass.setup_recording(recording, output_folder, verbose=verbose)
---> 89 SorterClass.run_from_folder(output_folder, raise_error, verbose)
     90 if with_output:
     91     sorting = SorterClass.get_result_from_folder(output_folder)

File ~\.conda\envs\si_env\lib\site-packages\spikeinterface\sorters\basesorter.py:236, in BaseSorter.run_from_folder(cls, output_folder, raise_error, verbose)
    234 if has_error and raise_error:
    235     print(log['error_trace'])
--> 236     raise SpikeSortingError(
    237         f"Spike sorting failed. You can inspect the runtime trace in {output_folder}/spikeinterface_log.json")
    239 return run_time

SpikeSortingError: Spike sorting failed. You can inspect the runtime trace in _output_KS3/spikeinterface_log.json

Here’s the log:
spikeinterface_log.txt

Note that:

1. I’ve successfully ran mexGPUall.m
2. when I run mex -setup C++, I get: MEX configured to use ‘Microsoft Visual C++ 2015’ for C++ language compilation.
3. the kilosort GUI seems to work
4. one RED flag is when I run kilosort3_master.m, I get: errString = 'kilosort repo not found, or inaccessible.'

Thank you so much!

[phanhuynh]

Man, if I could buy you a beer through this computer, I would. Thank you so much!

[phanhuynh]

Good afternoon, gentlemen! Thank you again for all your support. I’ve had a few issues but now it seems to be running. I will follow up as soon as it finishes!

Docker: pulling image spikeinterface/kilosort3-compiled-base
Starting container
Installing spikeinterface from sources in spikeinterface/kilosort3-compiled-base
Installing dev spikeinterface from remote repository
Installing extra requirements: []
Running kilosort3 sorter inside spikeinterface/kilosort3-compiled-base