cupy.cuda.runtime.CUDARuntimeError: cudaErrorNoDevice: no CUDA-capable device is detected

I am using AWS p3 type 2xlarge instance type with following GPU details: root@awsml04:~# nvcc -V nvcc: NVIDIA ® Cuda compiler driver Copyright © 2005-2017 NVIDIA Corporation Built on Fri_Sep__1_21:08:03_CDT_2017 Cuda compilation tools, release 9.0, V9.0.176

• Conditions (you can just paste the output of python -c 'import cupy; cupy.show_config()')
  • CuPy version cupy==5.3.0
  • OS/Platform Ubuntu 16.04 x86_64
  • CUDA version CUDA 9.0
  • cuDNN version: 7.3.1 -NCCL version: 2.3.5
• Code to reproduce
• Error messages, stack traces, or logs :

Exception in main training loop: cudaErrorNoDevice: no CUDA-capable device is detected
Traceback (most recent call last):
  File "/root/.see-master/lib/python3.5/site-packages/chainer/training/trainer.py", line 302, in run
    entry.extension(self)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/root/.see-master/lib/python3.5/site-packages/chainer/reporter.py", line 98, in scope
    yield
  File "/root/.see-master/lib/python3.5/site-packages/chainer/training/trainer.py", line 299, in run
    update()
  File "/root/.see-master/lib/python3.5/site-packages/chainer/training/updater.py", line 223, in update
    self.update_core()
  File "/root/.see-master/lib/python3.5/site-packages/chainer/training/updaters/multiprocess_parallel_updater.py", line 195, in update_core
    self.setup_workers()
  File "/root/.see-master/lib/python3.5/site-packages/chainer/training/updaters/multiprocess_parallel_updater.py", line 186, in setup_workers
    with cuda.Device(self._devices[0]):
  File "cupy/cuda/device.pyx", line 106, in cupy.cuda.device.Device.__enter__
  File "cupy/cuda/runtime.pyx", line 164, in cupy.cuda.runtime.getDevice
  File "cupy/cuda/runtime.pyx", line 136, in cupy.cuda.runtime.check_status
Will finalize trainer extensions and updater before reraising the exception.
Traceback (most recent call last):
  File "chainer/train_svhn.py", line 258, in <module>
    trainer.run()
  File "/root/.see-master/lib/python3.5/site-packages/chainer/training/trainer.py", line 313, in run
    six.reraise(*sys.exc_info())
  File "/usr/lib/python3.5/site-packages/six.py", line 693, in reraise
    raise value
  File "/root/.see-master/lib/python3.5/site-packages/chainer/training/trainer.py", line 302, in run
    entry.extension(self)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/root/.see-master/lib/python3.5/site-packages/chainer/reporter.py", line 98, in scope
    yield
  File "/root/.see-master/lib/python3.5/site-packages/chainer/training/trainer.py", line 299, in run
    update()
  File "/root/.see-master/lib/python3.5/site-packages/chainer/training/updater.py", line 223, in update
    self.update_core()
  File "/root/.see-master/lib/python3.5/site-packages/chainer/training/updaters/multiprocess_parallel_updater.py", line 195, in update_core
    self.setup_workers()
  File "/root/.see-master/lib/python3.5/site-packages/chainer/training/updaters/multiprocess_parallel_updater.py", line 186, in setup_workers
    with cuda.Device(self._devices[0]):
  File "cupy/cuda/device.pyx", line 106, in cupy.cuda.device.Device.__enter__
  File "cupy/cuda/runtime.pyx", line 164, in cupy.cuda.runtime.getDevice
  File "cupy/cuda/runtime.pyx", line 136, in cupy.cuda.runtime.check_status
cupy.cuda.runtime.CUDARuntimeError: cudaErrorNoDevice: no CUDA-capable device is detected

[crcrpar]

Could you check the number of GPUs your instance have?

The reason I ask this is that MultiprocessParallelUpdater assumes that a machine has multiple GPUs.

This is an implementation of Updater that uses multiple GPUs with multi-process data parallelism. It uses Nvidia NCCL for communication between multiple GPUs.

[fiarabbit]

@harshalcse https://groups.google.com/forum/#!forum/chainer You can find this link (and other helpful link) at https://github.com/chainer/chainer (https://github.com/chainer/chainer/blob/master/README.md)