[Bug] spleeter-gpu is unusable out of the box

Description

I tried, once again, to get spleeter running on the GPU, but I just can’t do it. I never directly worked with TensorFlow or any other ML software package, so I can’t even begin to debug this.

Step to reproduce

1. Install CUDA 10.2 on a fresh, fully up-to-date Windows 10 installation
2. Install Miniconda
3. Install spleeter-gpu using conda install -c conda-forge spleeter-gpu in the Anaconda Prompt
4. Run spleeter separate -o Spleeter -m -p spleeter:2stems-16kHz -i "audiofile.m4a"

Output

Traceback (most recent call last):
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
    return fn(*args)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node conv2d_7/Conv2D}}]]
         [[strided_slice_25/_309]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node conv2d_7/Conv2D}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\user\miniconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\user\miniconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\Miniconda3\Scripts\spleeter.exe\__main__.py", line 7, in <module>
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\__main__.py", line 54, in entrypoint
    main(sys.argv)
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\__main__.py", line 46, in main
    entrypoint(arguments, params)
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\commands\separate.py", line 43, in entrypoint
    synchronous=False
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\separator.py", line 123, in separate_to_file
    sources = self.separate(waveform)
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\separator.py", line 89, in separate
    'audio_id': ''})
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\contrib\predictor\predictor.py", line 77, in __call__
    return self._session.run(fetches=self.fetch_tensors, feed_dict=feed_dict)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
    run_metadata_ptr)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run
    run_metadata)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[node conv2d_7/Conv2D (defined at c:\users\user\miniconda3\lib\site-packages\spleeter\utils\estimator.py:71) ]]
         [[strided_slice_25/_309]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[node conv2d_7/Conv2D (defined at c:\users\user\miniconda3\lib\site-packages\spleeter\utils\estimator.py:71) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'conv2d_7/Conv2D':
  File "c:\users\user\miniconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\user\miniconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\Miniconda3\Scripts\spleeter.exe\__main__.py", line 7, in <module>
    sys.exit(entrypoint())
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\__main__.py", line 54, in entrypoint
    main(sys.argv)
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\__main__.py", line 46, in main
    entrypoint(arguments, params)
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\commands\separate.py", line 43, in entrypoint
    synchronous=False
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\separator.py", line 123, in separate_to_file
    sources = self.separate(waveform)
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\separator.py", line 86, in separate
    predictor = self._get_predictor()
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\separator.py", line 58, in _get_predictor
    self._predictor = to_predictor(estimator)
  File "c:\users\user\miniconda3\lib\site-packages\spleeter\utils\estimator.py", line 71, in to_predictor
    return predictor.from_saved_model(latest)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\contrib\predictor\predictor_factories.py", line 153, in from_saved_model
    config=config)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\contrib\predictor\saved_model_predictor.py", line 153, in __init__
    loader.load(self._session, tags.split(','), export_dir)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\saved_model\loader_impl.py", line 269, in load
    return loader.load(sess, tags, import_scope, **saver_kwargs)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\saved_model\loader_impl.py", line 422, in load
    **saver_kwargs)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\saved_model\loader_impl.py", line 352, in load_graph
    meta_graph_def, import_scope=import_scope, **saver_kwargs)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\training\saver.py", line 1473, in _import_meta_graph_with_return_elements
    **kwargs))
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\framework\meta_graph.py", line 857, in import_scoped_meta_graph_with_return_elements
    return_elements=return_elements)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\framework\importer.py", line 443, in import_graph_def
    _ProcessNewOps(graph)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\framework\importer.py", line 236, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3751, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3751, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3641, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "c:\users\user\miniconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

Environment

OS	Windows 10
Installation type	Conda
RAM available	32 GB
Hardware spec	Ryzen 9 3900x / RTX 2070 Super

[aidv]

Here are some possible fixes:

1. Check CUDA version. You should install 10.1
2. Check FFMPEG installation
3. Try running spleeter with python -m spleeter.......
4. Split audio files into several pieces if the whole audio file fails

[aidv]

@Laurent172 really? Nice. But I can’t get it to work. Is it for the latest Spleeter or the old one with the label cf202003?

I am having so much issues it’s insane.

Edit: Ok so I uninstalled Anaconda, removed any Anaconda related files in c:/users/username/. I then reinstalled Anaconda, created a new conda env, and installed conda install -c conda-forge/label/cf202003 spleeter-gpu.

This allows me to process a ~3 minute file in less than 10 seconds using my RTX2080.

This is EXACTLY what I want.

I’m going to see how it performs on my production server when people are using my service.