[Bug] cuDNN failed to initialize

Description

I know this has been discussed before, but maybe someone has a clear fix to this.

One of my machines doesn’t want to process any audio files and I get the following error message:

(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node conv2d_7/Conv2D}}]]

Step to reproduce

1. Installed using conda
2. Run as python -m
3. Got above error

Output

Traceback (most recent call last):
  File "c:\Users\user\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
    return fn(*args)
  File "c:\Users\user\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "c:\Users\user\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node conv2d_7/Conv2D}}]]
	 [[strided_slice_21/_307]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node conv2d_7/Conv2D}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\user\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\Users\user\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "c:\Users\user\Anaconda3\lib\site-packages\spleeter\__main__.py", line 58, in <module>
    entrypoint()
  File "c:\Users\user\Anaconda3\lib\site-packages\spleeter\__main__.py", line 54, in entrypoint
    main(sys.argv)
  File "c:\Users\user\Anaconda3\lib\site-packages\spleeter\__main__.py", line 46, in main
    entrypoint(arguments, params)
  File "c:\Users\user\Anaconda3\lib\site-packages\spleeter\commands\separate.py", line 45, in entrypoint
    synchronous=False
  File "c:\Users\user\Anaconda3\lib\site-packages\spleeter\separator.py", line 191, in separate_to_file
    sources = self.separate(waveform, audio_descriptor)
  File "c:\Users\user\Anaconda3\lib\site-packages\spleeter\separator.py", line 155, in separate
    return self.separate_tensorflow(waveform, audio_descriptor)
  File "c:\Users\user\Anaconda3\lib\site-packages\spleeter\separator.py", line 106, in separate_tensorflow
    'audio_id': audio_descriptor})
  File "c:\Users\user\Anaconda3\lib\site-packages\tensorflow\contrib\predictor\predictor.py", line 77, in __call__
    return self._session.run(fetches=self.fetch_tensors, feed_dict=feed_dict)
  File "c:\Users\user\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
    run_metadata_ptr)
  File "c:\Users\user\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "c:\Users\user\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run
    run_metadata)
  File "c:\Users\user\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node conv2d_7/Conv2D (defined at \site-packages\spleeter\utils\estimator.py:80) ]]
	 [[strided_slice_21/_307]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node conv2d_7/Conv2D (defined at \site-packages\spleeter\utils\estimator.py:80) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'conv2d_7/Conv2D':
  File "\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "\site-packages\spleeter\__main__.py", line 58, in <module>
    entrypoint()
  File "\site-packages\spleeter\__main__.py", line 54, in entrypoint
    main(sys.argv)
  File "\site-packages\spleeter\__main__.py", line 46, in main
    entrypoint(arguments, params)
  File "\site-packages\spleeter\commands\separate.py", line 45, in entrypoint
    synchronous=False
  File "\site-packages\spleeter\separator.py", line 191, in separate_to_file
    sources = self.separate(waveform, audio_descriptor)
  File "\site-packages\spleeter\separator.py", line 155, in separate
    return self.separate_tensorflow(waveform, audio_descriptor)
  File "\site-packages\spleeter\separator.py", line 103, in separate_tensorflow
    predictor = self._get_predictor()
  File "\site-packages\spleeter\separator.py", line 75, in _get_predictor
    self._predictor = to_predictor(estimator)
  File "\site-packages\spleeter\utils\estimator.py", line 80, in to_predictor
    return predictor.from_saved_model(latest)
  File "\site-packages\tensorflow\contrib\predictor\predictor_factories.py", line 153, in from_saved_model
    config=config)
  File "\site-packages\tensorflow\contrib\predictor\saved_model_predictor.py", line 153, in __init__
    loader.load(self._session, tags.split(','), export_dir)
  File "\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "\site-packages\tensorflow\python\saved_model\loader_impl.py", line 269, in load
    return loader.load(sess, tags, import_scope, **saver_kwargs)
  File "\site-packages\tensorflow\python\saved_model\loader_impl.py", line 422, in load
    **saver_kwargs)
  File "\site-packages\tensorflow\python\saved_model\loader_impl.py", line 352, in load_graph
    meta_graph_def, import_scope=import_scope, **saver_kwargs)
  File "\site-packages\tensorflow\python\training\saver.py", line 1473, in _import_meta_graph_with_return_elements
    **kwargs))
  File "\site-packages\tensorflow\python\framework\meta_graph.py", line 857, in import_scoped_meta_graph_with_return_elements
    return_elements=return_elements)
  File "\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "\site-packages\tensorflow\python\framework\importer.py", line 443, in import_graph_def
    _ProcessNewOps(graph)
  File "\site-packages\tensorflow\python\framework\importer.py", line 236, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "\site-packages\tensorflow\python\framework\ops.py", line 3751, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "\site-packages\tensorflow\python\framework\ops.py", line 3751, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "\site-packages\tensorflow\python\framework\ops.py", line 3641, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

Environment

OS	Windows 10
Installation type	Conda
RAM available	8 GB VRAM / 8 GB RAM
Hardware spec	RTX2080 / i7 960

[feliperyan]

For anyone having issues on Windows10:

Seemed like cuDNN was failing to init due to not being able to allocate the necessary memory on my GPU (rtx 2070 Super) upfront. Fix was to set a flag to allow incremental memory growth. Following script worked for me:

import tensorflow as tf

config = tf.ConfigProto()
config.gpu_options.allow_growth  = True
tf.Session(config=config) 

from spleeter.separator import Separator
from spleeter.audio.adapter import get_default_audio_adapter

separator = Separator('spleeter:2stems')
separator.separate_to_file('the-legacy.mp3', './')

[alreadytaikeune]

The older docker images do not work with tensorflow 1.15 due to cuda incompatibility. We will push newer images with updated dependencies.