Error message: CUBLAS_STATUS_NOT_INITIALIZED
See original GitHub issuenvcc --version
yields
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Oct_12_20:09:46_PDT_2020
Cuda compilation tools, release 11.1, V11.1.105
Build cuda_11.1.TC455_06.29190527_0
I installed jax with
pip install --upgrade pip
pip install --upgrade jax[cuda111] -f https://storage.googleapis.com/jax-releases/jax_releases.html
My code is
import jax.numpy as jnp
from jax import grad, jit, vmap
from jax import random
from time import time
key = random.PRNGKey(0)
def apply_matrix(v):
global key
mat = random.normal(key, (150, 100))
return jnp.dot(mat, v)
def batched_apply_matrix(v_batched):
return vmap(apply_matrix)(v_batched)
v_batched = random.normal(key, (10, 100))
batched_apply_matrix(v_batched).block_until_ready()
The program output is
2021-06-26 17:01:46.891583: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2021-06-26 17:01:46.891610: F external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:113] Check failed: stream->parent()->GetBlasGemmAlgorithms(&algorithms)
Aborted (core dumped)
Issue Analytics
- State:
- Created 2 years ago
- Reactions:1
- Comments:11 (1 by maintainers)
Top Results From Across the Web
CUBLAS_STATUS_NOT_INITIA...
I have been trying to run librispeech example with some gsm data. Although my cuda(version 8.0) is working alright. nnet-train-simple is throwing me...
Read more >CUBLAS_STATUS_NOT_INITIA...
This error might be raised, if you are running out of memory and cublas fails to create the handle, so try to reduce...
Read more >python - failed to create cublas handle ... - Stack Overflow
failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED. Error ... Error messages: Profiling failure on CUDNN engine 1#TC: UNKNOWN: ...
Read more >CUDA error: CUBLAS_STATUS_NOT_INITIALIZED - Reddit
As soon I try to generate an image I'm getting a RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)` ...
Read more >cublas_status_not_initialized when calling cublascreate(handle)
I got the issue 'RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED ... Hi, I tried to add some other embeddings in your BertEmbedding source code and ......
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Following #5380, when I tried out with
export XLA_PYTHON_CLIENT_PREALLOCATE=false
, it stopped crashing.When using Haiku together with TensorFlow datasets (tfds), I find it helpful to set
TF_FORCE_GPU_ALLOW_GROWTH=true
such that TF wouldn’t preallocate loads of GPU memory.