mmcv-full not compiled when building inside docker

Checklist

I know this error has been brought up several times

https://github.com/open-mmlab/mmdetection/issues/2686 https://github.com/open-mmlab/mmdetection/issues/4075

But Iv checked all solutions, all didnt work out for me.

I am building mmcv in docker I am using this pytorch image: FROM nvcr.io/nvidia/pytorch:20.11-py3 (which has pytorch 1.8.0, cuda 11.1.0)

I tried this

FROM nvcr.io/nvidia/pytorch:20.11-py3
........(omit other comands which are irrelevant)...........

RUN git clone https://github.com/open-mmlab/mmcv.git && \
cd mmcv && \
MMCV_WITH_OPS=1 pip install -e .

FROM nvcr.io/nvidia/pytorch:20.11-py3
pip install mmcv-full==1.3.8 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html

and many more versions, both didnt work… according to mmcv installation guide… mmcv-full 1.3.8 should complied with pytorch 1.3.8 cuda 11.1.0. isnt it?

I ran out of ideas… stuck here for few days… can someone please help me out… thanks

[BrianPugh]

fwiw, I was able to resolve this (while still using buildkit) by adding the following to my dockerfile (before installing mmcv)

ARG TORCH_CUDA_ARCH_LIST="7.5;6.1"
ENV FORCE_CUDA="1"

you can specify whatever compute capabiliies you want based on the hardware you are going to be running: https://developer.nvidia.com/cuda-gpus

[lingcong-k]

@zhouzaida Thanks… I found the issue.

so if anybody else facing the same issue. check two things

1. is the default runtime set to nvidia or not (under /etc/docker/daemon.json)

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    },
    "default-runtime": "nvidia"
}

1. if u r building docker using DOCKER_BUILDKIT it has issue of preventing access to nvidia runtime https://github.com/moby/buildkit/issues/1800 so dont use it