[BUG] Cannot Install NVTabular along PyTorch 1.9

Hi Team, I have been trying to run through the examples and wanted to setup my local environment for it:

Steps/Code to reproduce bug

• I install NVTabular using conda install -c nvidia -c rapidsai -c numba -c conda-forge nvtabular python=3.7 cudatoolkit=11.0
• Following which PyTorch 1.9 wouldn’t install, I follow the instructions from the PyTorch website for the same: conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch

Based on my previous encounters, I know PyTorch 1.9 doesn’t like Python <=3.8 due to a torchvision dependecy which is still being looked into, and it appears NVTabular won’t install with Python 3.9?

I tried installing PyTorch 1.7 however, it seems that the required version is 1.9 for making the examples work?

Apologies if I’m missing something straightforward here, I’ve tried different iterations of making the install work but I’m still stuck at getting things up and running.

Could you please point me to the right steps of getting everything up and running?

Thanks in advance for your help!

Aha! Link: https://nvaiinfra.aha.io/features/MERLIN-504

[NegatioN]

Bump and tag @rnyak: I guess the major issue is that [conda] Cudf is compatible with Cuda 11.0 & 11.2, while [conda] Pytorch[1.10] is compatible with cuda 11.3. Is there no option for Nvidia to build Cudf against all versions of cuda from 11.0 to 11.4 and release this in the conda repo?

That wouldn’t help for potential incompatabilities in the NVTabular & Transformers4Rec repos that might arise with each bump of Pytorch, but shouldn’t it make the build-process a lot smoother for pleb end-users like me? And it seems like a nice feature to have, unless it’s utterly impossible.

Edit: Seems like there’s already a thread for this going on in Cudf https://github.com/rapidsai/cudf/issues/8510

[benfred]

@pintonos - cudf doesn’t support cuda 10.2, and will require cuda 11.0+ https://github.com/rapidsai/cudf#cudagpu-requirements . nvtabular itself is pretty easy to install (you can install without cudf by going pip install nvtabular - but this will mean running nvtabular only on the CPU, and remove gpu support). The problem is getting cudf installed.

Can you try the docker container to see if that works for you (docker pull nvcr.io/nvidia/merlin/merlin-pytorch-training:21.09). This container includes cuda 11.4 and user mode drivers so should work from your host machine, even if that only has cuda 10.2 installed