Generated images are completely black?! 😵 What am I doing wrong?

Hello, I am on Windows 10, and my gpu is a PNY Nvidia GTX 1660 TI 6 Gb. I installed V-Diffusion like so:

• conda create --name v-diffusion python=3.8
• conda activate v-diffusion
• conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch (as per Pytorch website instructions)
• pip install requests tqdm

The problem is that when I launch the cfg_sample.py or clip_sample.py command lines, the generated images are completely black, although the inference process seems to run nicely and without errors.

Things I’ve tried:

• installing previous pytorch version with conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
• removing V-Diffusion conda environment completely and recreating it anew
• uninstalling nvidia drivers and performing a new clean driver install (I tried both Nvidia Studio drivers and Nvidia Game Ready drivers)
• uninstalling and reinstalling Conda completely

But nothing helped… and at this point I don’t know what else to try…

The only interesting piece of information I could gather is that for some reason this problem also happens with another text-to-image project called Big Sleep where similar to V-Diffusion the inference process appears to run correctly but the generated images are all black.

I think there must be some simple detail I’m overlooking… which it’s making me go insane… 😵 Please let me know something if you think you can help! THANKS !

[illtellyoulater]

I fixed it by installing (with pip) a version of Torch that includes CUDA Toolkit v10.2, like this:

pip install torch==1.10.1+cu102 torchvision==0.11.2+cu102 -f https://download.pytorch.org/whl/torch_stable.html

With any CUDA Toolkit version higher than that it would not work… not sure if this strict requirement comes from my system’s particular configuration, or this project, or the CLIP project…

Anyway, using v10.2 was enough to make cfg_sample.py work, but not to make clic_sample.py work.

To also fix that I had to make the following change to clip.py, at line 118:

https://github.com/openai/CLIP/blob/main/clip/clip.py#L118-L123

where, before the if condition, I had to add:

name = "ViT-B/32"

in order to force this model. Otherwise, by using other models I would either receive:

• a CUDA OUT OF MEMORY error
• A CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling 'cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)
• or a RuntimeError: mat1 dim 1 must match mat2 dim 0

That’s it! Maybe it doesn’t work at 100%, but it works and I’m having fun with it already! 😃 Thank you for working on this!

[illtellyoulater]

Just a quick update. I think we can rule out the option of a faulty GPU, as I could finally make it work with BigSleep and a couple of other projects. In those cases the solution was installing the latest version of torch with pip, like so: pip3 install torch==1.11.0+cu115 torchvision==0.12.0+cu115 -f https://download.pytorch.org/whl/torch_stable.html

However this did not work for both v-diffusion and glide-text2im, which continue to generate NaN values. All I can say for now is I suspect it has to do with the clip library…