question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Error message: CUBLAS_STATUS_NOT_INITIALIZED

See original GitHub issue

nvcc --version yields

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Oct_12_20:09:46_PDT_2020
Cuda compilation tools, release 11.1, V11.1.105
Build cuda_11.1.TC455_06.29190527_0

I installed jax with

pip install --upgrade pip
pip install --upgrade jax[cuda111] -f https://storage.googleapis.com/jax-releases/jax_releases.html

My code is

import jax.numpy as jnp
from jax import grad, jit, vmap
from jax import random

from time import time
key = random.PRNGKey(0)


def apply_matrix(v):
	global key
	mat = random.normal(key, (150, 100))
	return jnp.dot(mat, v)

def batched_apply_matrix(v_batched):
	return vmap(apply_matrix)(v_batched)

v_batched = random.normal(key, (10, 100))
batched_apply_matrix(v_batched).block_until_ready()

The program output is

2021-06-26 17:01:46.891583: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2021-06-26 17:01:46.891610: F external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:113] Check failed: stream->parent()->GetBlasGemmAlgorithms(&algorithms) 
Aborted (core dumped)

Issue Analytics

  • State:open
  • Created 2 years ago
  • Reactions:1
  • Comments:11 (1 by maintainers)

github_iconTop GitHub Comments

8reactions
shailesh1729commented, Oct 23, 2021

Following #5380, when I tried out with export XLA_PYTHON_CLIENT_PREALLOCATE=false, it stopped crashing.

2reactions
xidulucommented, Feb 15, 2022

When using Haiku together with TensorFlow datasets (tfds), I find it helpful to set TF_FORCE_GPU_ALLOW_GROWTH=true such that TF wouldn’t preallocate loads of GPU memory.

Read more comments on GitHub >

github_iconTop Results From Across the Web

CUBLAS_STATUS_NOT_INITIA...
I have been trying to run librispeech example with some gsm data. Although my cuda(version 8.0) is working alright. nnet-train-simple is throwing me...
Read more >
CUBLAS_STATUS_NOT_INITIA...
This error might be raised, if you are running out of memory and cublas fails to create the handle, so try to reduce...
Read more >
python - failed to create cublas handle ... - Stack Overflow
failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED. Error ... Error messages: Profiling failure on CUDNN engine 1#TC: UNKNOWN: ...
Read more >
CUDA error: CUBLAS_STATUS_NOT_INITIALIZED - Reddit
As soon I try to generate an image I'm getting a RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)` ...
Read more >
cublas_status_not_initialized when calling cublascreate(handle)
I got the issue 'RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED ... Hi, I tried to add some other embeddings in your BertEmbedding source code and ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found