Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

train_mnist on Chainer 1.10 + Ubuntu 16.04 + Cuda V7.5

See original GitHub issue

When I use Chainer with Mnist, I got a weird error Value 'sm_61' is not defined for option 'gpu-architecture'. I installed Cuda using https://www.pugetsystems.com/labs/hpc/NVIDIA-CUDA-with-Ubuntu-16-04-beta-on-a-laptop-if-you-just-cannot-wait-775/ All the Cuda samples works, except the ones described by the blog post.

$ chainer-1.9.1/examples/mnist/train_mnist.py --gpu 0 2>&1 | tee logs.txt
/usr/local/lib/python2.7/dist-packages/chainer/cuda.py:87: UserWarning: cuDNN is not enabled.
Please reinstall chainer after you install cudnn
(see https://github.com/pfnet/chainer#installation).
  'cuDNN is not enabled.\n'
GPU: 0
# unit: 1000
# Minibatch-size: 100
# epoch: 20
Network type: simple

load MNIST dataset
Downloading train-images-idx3-ubyte.gz...
Done
Downloading train-labels-idx1-ubyte.gz...
Done
Downloading t10k-images-idx3-ubyte.gz...
Done
Downloading t10k-labels-idx1-ubyte.gz...
Done
Converting training data...
Done
Converting test data...
Done
Save output...
Done
Convert completed
epoch 1
Traceback (most recent call last):
  File "/home/matthieu/Programmes/chainer-1.9.1/examples/mnist/train_mnist.py", line 102, in <module>
    optimizer.update(model, x, t)
  File "/usr/local/lib/python2.7/dist-packages/chainer/optimizer.py", line 375, in update
    self.target.zerograds()
  File "/usr/local/lib/python2.7/dist-packages/chainer/link.py", line 530, in zerograds
    super(Chain, self).zerograds()
  File "/usr/local/lib/python2.7/dist-packages/chainer/link.py", line 318, in zerograds
    param.zerograd()
  File "/usr/local/lib/python2.7/dist-packages/chainer/variable.py", line 219, in zerograd
    self._grad.fill(0)
  File "cupy/core/core.pyx", line 344, in cupy.core.core.ndarray.fill (cupy/core/core.cpp:7636)
  File "cupy/core/core.pyx", line 353, in cupy.core.core.ndarray.fill (cupy/core/core.cpp:7581)
  File "cupy/core/core.pyx", line 1292, in cupy.core.core.elementwise_copy (cupy/core/core.cpp:49352)
  File "cupy/core/elementwise.pxi", line 768, in cupy.core.core.ufunc.__call__ (cupy/core/core.cpp:40183)
  File "cupy/util.pyx", line 36, in cupy.util.memoize.decorator.ret (cupy/util.cpp:1194)
  File "cupy/core/elementwise.pxi", line 576, in cupy.core.core._get_ufunc_kernel (cupy/core/core.cpp:36755)
  File "cupy/core/elementwise.pxi", line 32, in cupy.core.core._get_simple_elementwise_kernel (cupy/core/core.cpp:27237)
  File "cupy/core/carray.pxi", line 87, in cupy.core.core.compile_with_cache (cupy/core/core.cpp:26915)
  File "/usr/local/lib/python2.7/dist-packages/cupy/cuda/compiler.py", line 141, in compile_with_cache
    cubin = nvcc(source, options, arch)
  File "/usr/local/lib/python2.7/dist-packages/cupy/cuda/compiler.py", line 66, in nvcc
    _run_nvcc(cmd, root_dir)
  File "/usr/local/lib/python2.7/dist-packages/cupy/cuda/compiler.py", line 44, in _run_nvcc
    raise RuntimeError(msg)
RuntimeError: `nvcc` command returns non-zero exit status.
command: ['nvcc', '--cubin', '-arch', 'sm_61', '/tmp/tmpZRGwXz/kern.cu']
return-code: 1
stdout/stderr:
nvcc fatal   : Value 'sm_61' is not defined for option 'gpu-architecture'