ArrayMemoryError when using tridesclous

Hello team, I’ve attached the error log for tridesclous.
The key phrase is:

raise self._exception\nnumpy.core._exceptions._ArrayMemoryError: Unable to allocate 10.2 GiB for an array with shape (42923008, 64) and data type float32\n",
    "run_time": null

Any thoughts? Thank you very much! spikeinterface_log.txt

[phanhuynh]

Hi Sam - no worries, I know you’re super busy. Great idea though about using docker. Unfortunately, I get the error pasted below. @Can, thank you for all your ideas! I’ll try them if I can’t get docker to work. But to answer your first question, the raw file is 20.6GB.

Starting container
Installing spikeinterface==0.95.1 in spikeinterface/tridesclous-base
Installing extra requirements: ['neo']
Running tridesclous sorter inside spikeinterface/tridesclous-base
Stopping container
---------------------------------------------------------------------------
SpikeSortingError                         Traceback (most recent call last)
Input In [9], in <cell line: 17>()
      1 #####################################################
      2 ### Option 1: Run local sorter (recommended) ###########
      3 #####################################################
   (...)
     14 ### Option 2: run sorter directly from docker ###
     15 #####################################################
     16 start_time = time.time()
---> 17 sorting_TDC = ss.run_sorter(
     18     'tridesclous',
     19     recording_saved,
     20     output_folder= '_output_TDC',
     21     remove_existing_folder=True,
     22     detect_threshold=threshold,
     23     docker_image="spikeinterface/tridesclous-base", 
     24     verbose=True,
     25 )
     26 print("--- _output_TDC took %s seconds ---" % (time.time() - start_time))
     27 print(f'Tridesclous found {len(sorting_TDC.get_unit_ids())} units')

File ~\Anaconda3\envs\sienv\lib\site-packages\spikeinterface\sorters\runsorter.py:136, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, with_output, **sorter_params)
    134         else:
    135             container_image = singularity_image
--> 136     return run_sorter_container(
    137         container_image=container_image,
    138         mode=mode,
    139         **common_kwargs,
    140     )
    142 return run_sorter_local(**common_kwargs)

File ~\Anaconda3\envs\sienv\lib\site-packages\spikeinterface\sorters\runsorter.py:548, in run_sorter_container(sorter_name, recording, mode, container_image, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, extra_requirements, **sorter_params)
    546 if run_error:
    547     if raise_error:
--> 548         raise SpikeSortingError(
    549             f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
    550 else:
    551     if with_output:

SpikeSortingError: Spike sorting in docker failed with the following error:
b'Traceback (most recent call last):\n  File "/Users/huynh/spike_epsztein/Phan/proj_nexus_v2/in_container_sorter_script.py", line 17, in <module>\n    sorting = run_sorter_local(\n  File "/usr/local/lib/python3.8/site-packages/spikeinterface/sorters/runsorter.py", line 157, in run_sorter_local\n    output_folder = SorterClass.initialize_folder(\n  File "/usr/local/lib/python3.8/site-packages/spikeinterface/sorters/basesorter.py", line 123, in initialize_folder\n    shutil.rmtree(str(output_folder))\n  File "/usr/local/lib/python3.8/shutil.py", line 718, in rmtree\n    _rmtree_safe_fd(fd, path, onerror)\n  File "/usr/local/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd\n    _rmtree_safe_fd(dirfd, fullname, onerror)\n  File "/usr/local/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd\n    _rmtree_safe_fd(dirfd, fullname, onerror)\n  File "/usr/local/lib/python3.8/shutil.py", line 675, in _rmtree_safe_fd\n    onerror(os.unlink, fullname, sys.exc_info())\n  File "/usr/local/lib/python3.8/shutil.py", line 673, in _rmtree_safe_fd\n    os.unlink(entry.name, dir_fd=topfd)\nPermissionError: [Errno 13] Permission denied: \'all_peaks.raw\'\n'

[phanhuynh]

Hi Can, thank you for the quick response. Yes, this same data set worked with Kilosort3, Spyking Circus, and Iron Cluster.
I’ll check if the ram is maxed out when I run it. I’ll also loked into Swap. Thanks!