KFServing instructions fail on Docker build

Context

I’m attempting to follow the instructions at https://github.com/pytorch/serve/tree/master/kubernetes/kfserving but they fail on the second step:

DOCKER_BUILDKIT=1 docker build --no-cache --file Dockerfile_kf.dev -t <docker image name> .

(This is after substituting a suitable name for <docker image name>, of course.)

• torchserve version: Latest master, commit hash 3bc25565bcc0ceebc44fb9396ef15a933b8d45e6
• torch version: n/a (but the Docker build is attempting to install 1.7.1)
• torchvision version [if any]: n/a
• torchtext version [if any]: n/a
• torchaudio version [if any]: n/a
• java version:

openjdk 11.0.9.1 2020-11-04
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9.1+1)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.9.1+1, mixed mode)

• Operating System and version: MacOS 10.15.7 building a Docker with ARG BASE_IMAGE=ubuntu:18.04

Your Environment

• Installed using source? [yes/no]: yes
• Are you planning to deploy it using docker container? [yes/no]: I’m trying to
• Is it a CPU or GPU environment?: CPU
• Using a default/custom handler? not relevant, but: serve/examples/image_classifier/mnist/mnist_handler.py
• What kind of model is it e.g. vision, text, audio?: not relevant, but mnist (image classifier)
• Are you planning to use local models from model-store or public url being used e.g. from S3 bucket etc.? [If public url then provide link.]: attempting local
• Provide config.properties, logs [ts.log] and parameters used for model registration/update APIs: n/a - haven’t even gotten to that step
• Link to your project [if any]: n/a

Expected Behavior

Docker image created

Current Behavior

Docker image creation fails (see log below) after failing to install pygit2

Possible Solution

This mimicks a bug that used to exist with ts_scripts/install_dependencies.py. On Mac, running ts_scripts/install_dependencies.py --environment=dev installs pygit2 correctly; this command line is correctly updated in the Dockerfile, but still fails.

Attempting to change it to conda install pygit2 -c conda-forge or pip install pygit2 does not appear to help.

Steps to Reproduce

1. Follow the steps at https://github.com/pytorch/serve/tree/master/kubernetes/kfserving
2. The second step, DOCKER_BUILDKIT=1 docker build --no-cache --file Dockerfile_kf.dev -t <docker image name> ., fails, showing the log below.

Failure Logs [if any]

 => ERROR [dev-image 1/2] RUN if [ "cpu" = "gpu" ]; then export USE_CUDA=1; fi     && git clone -b master https://github.com/kubeflow/kfserving.git     && pip install -e .  136.8s
------
 > [dev-image 1/2] RUN if [ "cpu" = "gpu" ]; then export USE_CUDA=1; fi     && git clone -b master https://github.com/kubeflow/kfserving.git     && pip install -e ./kfserving/python/kfserving     && git clone https://github.com/pytorch/serve.git     && cd serve     && git checkout master     && python ts_scripts/install_dependencies.py --environment=dev --cuda latest     && python ts_scripts/install_from_src.py     && pip install captum     && pip install transformers    && pip install .     && useradd -m model-server     && mkdir -p /home/model-server/tmp     && cp kubernetes/kfserving/kf_predictor_docker/dockerd-entrypoint_kf.sh /usr/local/bin/dockerd-entrypoint_kf.sh     && chmod +x /usr/local/bin/dockerd-entrypoint_kf.sh     && chown -R model-server /home/model-server     && cp -R kubernetes/kfserving/kfserving_wrapper /home/model-server/kfserving_wrapper     && cp kubernetes/kfserving/kf_predictor_docker/config_kf.properties /home/model-server/config.properties     && mkdir /home/model-server/model-store && chown -R model-server /home/model-server/model-store:
#10 0.270 Cloning into 'kfserving'...
#10 9.051 Obtaining file:///kfserving/python/kfserving
#10 9.656 Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.6/dist-packages (from kfserving==0.5.0) (51.0.0)
#10 10.91 Collecting adal>=1.2.2
#10 11.00   Downloading adal-1.2.5-py2.py3-none-any.whl (55 kB)
#10 11.76 Collecting argparse>=1.4.0
#10 11.78   Downloading argparse-1.4.0-py2.py3-none-any.whl (23 kB)
#10 11.78 Collecting azure-storage-blob<=2.1.0,>=1.3.0
#10 11.80   Downloading azure_storage_blob-2.1.0-py2.py3-none-any.whl (88 kB)
#10 11.93 Collecting azure-common>=1.1.5
#10 11.96   Downloading azure_common-1.1.26-py2.py3-none-any.whl (12 kB)
#10 11.97 Collecting azure-storage-common~=2.1
#10 11.98   Downloading azure_storage_common-2.1.0-py2.py3-none-any.whl (47 kB)
#10 11.99 Collecting certifi>=14.05.14
#10 12.01   Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
#10 12.02 Collecting cryptography>=1.1.0
#10 12.04   Downloading cryptography-3.3.1-cp36-abi3-manylinux2010_x86_64.whl (2.6 MB)
#10 12.61 Collecting cffi>=1.12
#10 12.63   Downloading cffi-1.14.4-cp36-cp36m-manylinux1_x86_64.whl (401 kB)
#10 12.69 Collecting google-cloud-storage>=1.31.0
#10 12.84   Downloading google_cloud_storage-1.33.0-py2.py3-none-any.whl (92 kB)
#10 13.15 Collecting google-auth<2.0dev,>=1.11.0
#10 13.17   Downloading google_auth-1.23.0-py2.py3-none-any.whl (114 kB)
#10 13.40 Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.6/dist-packages (from kfserving==0.5.0) (51.0.0)
#10 13.41 Collecting cachetools<5.0,>=2.0.0
#10 13.43   Downloading cachetools-4.1.1-py3-none-any.whl (10 kB)
#10 13.44 Collecting google-cloud-core<2.0dev,>=1.4.1
#10 13.45   Downloading google_cloud_core-1.4.4-py2.py3-none-any.whl (27 kB)
#10 13.56 Collecting google-api-core<2.0.0dev,>=1.21.0
#10 13.57   Downloading google_api_core-1.23.0-py2.py3-none-any.whl (91 kB)
#10 13.59 Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.6/dist-packages (from kfserving==0.5.0) (51.0.0)
#10 14.15 Collecting google-resumable-media<2.0dev,>=1.0.0
#10 14.16   Downloading google_resumable_media-1.1.0-py2.py3-none-any.whl (75 kB)
#10 14.24 Collecting google-crc32c<2.0dev,>=1.0
#10 14.25   Downloading google_crc32c-1.0.0-cp36-cp36m-manylinux2010_x86_64.whl (39 kB)
#10 14.28 Collecting googleapis-common-protos<2.0dev,>=1.6.0
#10 14.30   Downloading googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
#10 14.33 Collecting kubernetes>=12.0.0
#10 14.35   Downloading kubernetes-12.0.1-py2.py3-none-any.whl (1.7 MB)
#10 14.47 Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.6/dist-packages (from kfserving==0.5.0) (51.0.0)
#10 14.77 Collecting minio>=4.0.9
#10 14.78   Downloading minio-7.0.0-py3-none-any.whl (72 kB)
#10 14.85 Collecting configparser
#10 14.87   Downloading configparser-5.0.1-py3-none-any.whl (22 kB)
#10 14.89 Collecting numpy>=1.17.3
#10 14.91   Downloading numpy-1.19.4-cp36-cp36m-manylinux2010_x86_64.whl (14.5 MB)
#10 15.53 Collecting protobuf>=3.12.0
#10 15.54   Downloading protobuf-3.14.0-cp36-cp36m-manylinux1_x86_64.whl (1.0 MB)
#10 15.58 Collecting pyasn1-modules>=0.2.1
#10 15.60   Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
#10 15.76 Collecting pyasn1<0.5.0,>=0.4.6
#10 15.78   Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
#10 15.79 Collecting pycparser
#10 15.80   Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
#10 15.82 Collecting PyJWT>=1.0.0
#10 15.83   Downloading PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
#10 15.85 Collecting python_dateutil>=2.5.3
#10 15.87   Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
#10 15.90 Collecting pytz
#10 15.92   Downloading pytz-2020.4-py2.py3-none-any.whl (509 kB)
#10 15.95 Collecting pyyaml>=3.12
#10 15.96   Downloading PyYAML-5.3.1.tar.gz (269 kB)
#10 16.32 Collecting requests>=2.0.0
#10 16.34   Downloading requests-2.25.0-py2.py3-none-any.whl (61 kB)
#10 16.52 Collecting chardet<4,>=3.0.2
#10 16.59   Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
#10 16.60 Collecting idna<3,>=2.5
#10 16.62   Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
#10 16.63 Collecting requests-oauthlib
#10 16.65   Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
#10 16.72 Collecting oauthlib>=3.0.0
#10 16.74   Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
#10 16.76 Collecting rsa<5,>=3.1.4
#10 16.78   Downloading rsa-4.6-py3-none-any.whl (47 kB)
#10 16.83 Collecting six>=1.15
#10 16.85   Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
#10 16.86 Collecting table_logger>=0.3.5
#10 16.88   Downloading table_logger-0.3.6-py3-none-any.whl (14 kB)
#10 16.89 Collecting tornado>=6.0.0
#10 16.91   Downloading tornado-6.1-cp36-cp36m-manylinux2010_x86_64.whl (427 kB)
#10 16.94 Collecting urllib3>=1.15.1
#10 16.95   Downloading urllib3-1.26.2-py2.py3-none-any.whl (136 kB)
#10 17.00 Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0
#10 17.01   Downloading websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
#10 17.05 Building wheels for collected packages: pyyaml
#10 17.05   Building wheel for pyyaml (setup.py): started
#10 17.43   Building wheel for pyyaml (setup.py): finished with status 'done'
#10 17.43   Created wheel for pyyaml: filename=PyYAML-5.3.1-cp36-cp36m-linux_x86_64.whl size=44620 sha256=ea171036d0a403b963f1e9a352d2fd2a2893dbe5ae6e60ddec428c1aaf430db1
#10 17.43   Stored in directory: /root/.cache/pip/wheels/e5/9d/ad/2ee53cf262cba1ffd8afe1487eef788ea3f260b7e6232a80fc
#10 17.44 Successfully built pyyaml
#10 17.62 Installing collected packages: six, pyasn1, urllib3, rsa, pycparser, pyasn1-modules, protobuf, idna, chardet, certifi, cachetools, requests, pytz, googleapis-common-protos, google-auth, cffi, python-dateutil, oauthlib, google-crc32c, google-api-core, cryptography, azure-common, websocket-client, requests-oauthlib, pyyaml, PyJWT, numpy, google-resumable-media, google-cloud-core, configparser, azure-storage-common, tornado, table-logger, minio, kubernetes, google-cloud-storage, azure-storage-blob, argparse, adal, kfserving
#10 23.07   Running setup.py develop for kfserving
#10 23.50 Successfully installed PyJWT-1.7.1 adal-1.2.5 argparse-1.4.0 azure-common-1.1.26 azure-storage-blob-2.1.0 azure-storage-common-2.1.0 cachetools-4.1.1 certifi-2020.12.5 cffi-1.14.4 chardet-3.0.4 configparser-5.0.1 cryptography-3.3.1 google-api-core-1.23.0 google-auth-1.23.0 google-cloud-core-1.4.4 google-cloud-storage-1.33.0 google-crc32c-1.0.0 google-resumable-media-1.1.0 googleapis-common-protos-1.52.0 idna-2.10 kfserving kubernetes-12.0.1 minio-7.0.0 numpy-1.19.4 oauthlib-3.1.0 protobuf-3.14.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 python-dateutil-2.8.1 pytz-2020.4 pyyaml-5.3.1 requests-2.25.0 requests-oauthlib-1.3.0 rsa-4.6 six-1.15.0 table-logger-0.3.6 tornado-6.1 urllib3-1.26.2 websocket-client-0.57.0
#10 23.65 Cloning into 'serve'...
#10 31.52 Already on 'master'
#10 31.52 Your branch is up to date with 'origin/master'.
#10 31.58 sh: 1: nvidia-smi: not found
#10 31.58 sh: 1: sudo: not found
#10 31.58 sh: 1: sudo: not found
#10 31.58 sh: 1: conda: not found
#10 32.35 Collecting torch==1.6.0
#10 33.45   Downloading torch-1.6.0-cp36-cp36m-manylinux1_x86_64.whl (748.8 MB)
#10 64.85 Killed
#10 65.56 Requirement already satisfied: pip in /usr/local/lib/python3.6/dist-packages (20.3.1)
#10 67.61 Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (51.0.0)
#10 76.33 Requirement already satisfied: wheel in /usr/local/lib/python3.6/dist-packages (from -r requirements/common.txt (line 3)) (0.36.1)
#10 77.15 Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from -r requirements/common.txt (line 4)) (2.25.0)
#10 81.11 Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 8)) (1.19.4)
#10 83.68 Requirement already satisfied: protobuf in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 10)) (3.14.0)
#10 85.86 Collecting captum
#10 85.97   Downloading captum-0.3.0-py3-none-any.whl (5.7 MB)
#10 86.70 Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 8)) (1.19.4)
#10 87.10 Collecting future
#10 87.12   Downloading future-0.18.2.tar.gz (829 kB)
#10 87.72 Collecting grpcio
#10 87.74   Downloading grpcio-1.34.0-cp36-cp36m-manylinux2014_x86_64.whl (4.0 MB)
#10 88.40 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 88.40 Collecting grpcio-tools
#10 88.42   Downloading grpcio_tools-1.34.0-cp36-cp36m-manylinux2014_x86_64.whl (2.5 MB)
#10 88.67 Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from grpcio-tools->-r requirements/developer.txt (line 11)) (51.0.0)
#10 88.68 Requirement already satisfied: protobuf in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 10)) (3.14.0)
#10 88.74 Collecting mock
#10 88.76   Downloading mock-4.0.3-py3-none-any.whl (28 kB)
#10 88.78 Collecting nvgpu
#10 88.79   Downloading nvgpu-0.9.0-py2.py3-none-any.whl (9.4 kB)
#10 89.57 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 89.64 Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from -r requirements/common.txt (line 4)) (2.25.0)
#10 89.82 Collecting packaging
#10 89.84   Downloading packaging-20.7-py2.py3-none-any.whl (35 kB)
#10 90.03 Collecting pillow
#10 90.05   Downloading Pillow-8.0.1-cp36-cp36m-manylinux1_x86_64.whl (2.2 MB)
#10 90.24 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 90.24 Collecting psutil
#10 90.26   Downloading psutil-5.7.3.tar.gz (465 kB)
#10 90.93 Collecting pygit2
#10 90.96   Downloading pygit2-1.4.0-cp36-cp36m-manylinux2010_x86_64.whl (3.0 MB)
#10 91.29 Requirement already satisfied: cffi>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from pygit2->-r requirements/developer.txt (line 13)) (1.14.4)
#10 91.30 Collecting pylint==2.6.0
#10 91.31   Downloading pylint-2.6.0-py3-none-any.whl (325 kB)
#10 91.85 Collecting pytest
#10 91.86   Downloading pytest-6.1.2-py3-none-any.whl (272 kB)
#10 92.36 Collecting pytest-cov
#10 92.38   Downloading pytest_cov-2.10.1-py2.py3-none-any.whl (19 kB)
#10 93.17 Collecting pytest-mock
#10 93.19   Downloading pytest_mock-3.3.1-py3-none-any.whl (11 kB)
#10 93.27 Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->-r requirements/common.txt (line 4)) (3.0.4)
#10 93.27 Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->-r requirements/common.txt (line 4)) (2.10)
#10 93.28 Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->-r requirements/common.txt (line 4)) (1.26.2)
#10 93.28 Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->-r requirements/common.txt (line 4)) (2020.12.5)
#10 93.28 Collecting sentencepiece
#10 93.32   Downloading sentencepiece-0.1.94-cp36-cp36m-manylinux2014_x86_64.whl (1.1 MB)
#10 93.44 Collecting transformers==2.10.0
#10 93.46   Downloading transformers-2.10.0-py3-none-any.whl (660 kB)
#10 95.05 Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from -r requirements/common.txt (line 4)) (2.25.0)
#10 95.33 Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 8)) (1.19.4)
#10 95.35 Collecting ansi2html
#10 95.36   Downloading ansi2html-1.6.0-py3-none-any.whl (14 kB)
#10 95.37 Collecting arrow
#10 95.39   Downloading arrow-0.17.0-py2.py3-none-any.whl (50 kB)
#10 95.40 Requirement already satisfied: python-dateutil>=2.7.0 in /usr/local/lib/python3.6/dist-packages (from arrow->nvgpu->-r requirements/developer.txt (line 7)) (2.8.1)
#10 95.41 Collecting astroid<=2.5,>=2.4.0
#10 95.42   Downloading astroid-2.4.2-py3-none-any.whl (213 kB)
#10 95.45 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 95.74 Collecting attrs>=17.4.0
#10 95.75   Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
#10 95.79 Collecting cached-property
#10 95.81   Downloading cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
#10 95.82 Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi>=1.4.0->pygit2->-r requirements/developer.txt (line 13)) (2.20)
#10 95.83 Collecting coverage>=4.4
#10 95.85   Downloading coverage-5.3-cp36-cp36m-manylinux1_x86_64.whl (229 kB)
#10 95.88 Collecting dataclasses
#10 95.90   Downloading dataclasses-0.8-py3-none-any.whl (19 kB)
#10 95.91 Collecting filelock
#10 95.92   Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
#10 95.93 Collecting flask
#10 95.95   Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
#10 96.27 Collecting click>=5.1
#10 96.28   Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
#10 96.30 Collecting flask-restful
#10 96.32   Downloading Flask_RESTful-0.3.8-py2.py3-none-any.whl (25 kB)
#10 96.43 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 96.43 Requirement already satisfied: pytz in /usr/local/lib/python3.6/dist-packages (from flask-restful->nvgpu->-r requirements/developer.txt (line 7)) (2020.4)
#10 96.43 Collecting aniso8601>=0.82
#10 96.45   Downloading aniso8601-8.1.0-py2.py3-none-any.whl (44 kB)
#10 96.46 Collecting importlib-metadata>=0.12
#10 96.48   Downloading importlib_metadata-3.1.1-py3-none-any.whl (9.6 kB)
#10 96.65 Collecting iniconfig
#10 96.66   Downloading iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
#10 96.68 Collecting isort<6,>=4.2.5
#10 96.69   Downloading isort-5.6.4-py3-none-any.whl (98 kB)
#10 96.72 Collecting itsdangerous>=0.24
#10 96.73   Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
#10 96.75 Collecting Jinja2>=2.10.1
#10 96.76   Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
#10 96.85 Collecting lazy-object-proxy==1.4.*
#10 96.87   Downloading lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55 kB)
#10 96.90 Collecting MarkupSafe>=0.23
#10 96.91   Downloading MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl (27 kB)
#10 96.94 Collecting matplotlib
#10 96.95   Downloading matplotlib-3.3.3-cp36-cp36m-manylinux1_x86_64.whl (11.6 MB)
#10 98.11 Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 8)) (1.19.4)
#10 98.12 Requirement already satisfied: python-dateutil>=2.7.0 in /usr/local/lib/python3.6/dist-packages (from arrow->nvgpu->-r requirements/developer.txt (line 7)) (2.8.1)
#10 98.22 Collecting cycler>=0.10
#10 98.24   Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
#10 98.25 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 98.25 Collecting kiwisolver>=1.0.1
#10 98.27   Downloading kiwisolver-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (1.1 MB)
#10 98.42 Collecting mccabe<0.7,>=0.6
#10 98.43   Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
#10 98.44 Collecting pandas
#10 98.46   Downloading pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)
#10 99.12 Requirement already satisfied: python-dateutil>=2.7.0 in /usr/local/lib/python3.6/dist-packages (from arrow->nvgpu->-r requirements/developer.txt (line 7)) (2.8.1)
#10 99.12 Requirement already satisfied: pytz in /usr/local/lib/python3.6/dist-packages (from flask-restful->nvgpu->-r requirements/developer.txt (line 7)) (2020.4)
#10 99.13 Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from -r requirements/developer.txt (line 8)) (1.19.4)
#10 99.16 Collecting pluggy<1.0,>=0.12
#10 99.18   Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
#10 99.20 Collecting py>=1.8.2
#10 99.22   Downloading py-1.9.0-py2.py3-none-any.whl (99 kB)
#10 99.26 Collecting pynvml
#10 99.28   Downloading pynvml-8.0.4-py3-none-any.whl (36 kB)
#10 99.30 Collecting pyparsing>=2.0.2
#10 99.32   Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
#10 99.35 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 99.37 Collecting regex!=2019.12.17
#10 99.39   Downloading regex-2020.11.13-cp36-cp36m-manylinux2014_x86_64.whl (723 kB)
#10 99.47 Collecting sacremoses
#10 99.49   Downloading sacremoses-0.0.43.tar.gz (883 kB)
#10 99.84 Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/dist-packages (from grpcio->-r requirements/developer.txt (line 9)) (1.15.0)
#10 99.96 Collecting joblib
#10 99.98   Downloading joblib-0.17.0-py3-none-any.whl (301 kB)
#10 100.0 Collecting tabulate
#10 100.1   Downloading tabulate-0.8.7-py3-none-any.whl (24 kB)
#10 100.1 Collecting termcolor
#10 100.1   Downloading termcolor-1.1.0.tar.gz (3.9 kB)
#10 100.3 Collecting tokenizers==0.7.0
#10 100.4   Downloading tokenizers-0.7.0-cp36-cp36m-manylinux1_x86_64.whl (3.8 MB)
#10 100.6 Collecting toml>=0.7.1
#10 100.7   Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
#10 100.7 Collecting torch>=1.2
#10 100.7   Downloading torch-1.7.1-cp36-cp36m-manylinux1_x86_64.whl (776.8 MB)
#10 135.2 Killed
#10 135.3 sh: 1: sudo: not found
#10 135.3 sh: 1: sudo: not found
#10 135.3 sh: 1: sudo: not found
#10 135.3 sh: 1: sudo: not found
#10 135.3 sh: 1: sudo: not found
#10 135.3 sh: 1: sudo: not found
#10 135.6 Traceback (most recent call last):
#10 135.6   File "ts_scripts/install_from_src.py", line 62, in <module>
#10 135.6     from pygit2 import Repository
#10 135.6 ModuleNotFoundError: No module named 'pygit2'
------
executor failed running [/bin/sh -c if [ "$MACHINE_TYPE" = "gpu" ]; then export USE_CUDA=1; fi     && git clone -b ${BRANCH_NAME_KF} https://github.com/kubeflow/kfserving.git     && pip install -e ./kfserving/python/kfserving     && git clone https://github.com/pytorch/serve.git     && cd serve     && git checkout ${BRANCH_NAME}     && python ts_scripts/install_dependencies.py --environment=dev --cuda $CUDA_VERSION     && python ts_scripts/install_from_src.py     && pip install captum     && pip install transformers    && pip install .     && useradd -m model-server     && mkdir -p /home/model-server/tmp     && cp kubernetes/kfserving/kf_predictor_docker/dockerd-entrypoint_kf.sh /usr/local/bin/dockerd-entrypoint_kf.sh     && chmod +x /usr/local/bin/dockerd-entrypoint_kf.sh     && chown -R model-server /home/model-server     && cp -R kubernetes/kfserving/kfserving_wrapper /home/model-server/kfserving_wrapper     && cp kubernetes/kfserving/kf_predictor_docker/config_kf.properties /home/model-server/config.properties     && mkdir /home/model-server/model-store && chown -R model-server /home/model-server/model-store]: exit code: 1

[fbbradheintz]

@maaquib We still need to investigate why PT 1.6 and 1.7.1 installs are getting called twice in the docker setup

The 1.6.0 installation failed on an OOME; later, when installing Captum, there is another attempt (also failed due to OOME) to install PyTorch as a dependency, and it takes latest (1.7.1). This is not a separate bug, just a consequence of the out-of-memory failure.

[maaquib]

@fbbradheintz I meant installing sudo here

Similar to this one