Sentencepiece dependency causing docker build to fail

Environment info

• transformers version:3.4.0
• Platform:Ubuntu 18.04
• Python version: 3.7
• PyTorch version (GPU?): 1.7.0 no gpu
• Tensorflow version (GPU?): N/A
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Information

Question answering pipelines is the feature I am using. There is a problem with the sentencepiece dependency of transformers. It cannot find the correct package when installing and causing the build process to fail

The problem arises when using: Downloading the transformers library onto a docker container running ubuntu.

The tasks I am working on is: Uploadding a transformers script to AWS fargate

To reproduce

Steps to reproduce the behavior:

1. Create a project
2. Try to create docker container using dockerfile attached below

I have attached the relevant parts of my dockerfile below Dockerfile

FROM ubuntu:18.04

RUN mkdir /usr/app
WORKDIR /usr/app

# Add and install Python modules
COPY requirements.txt ./
RUN apt-get update
RUN apt-get -y install python3
RUN apt-get -y install python3-pip
RUN pip3 install virtualenv
ENV VIRTUAL_ENV=/venv
RUN virtualenv venv -p python3
ENV PATH="VIRTUAL_ENV/bin:$PATH"

RUN pip3 install transformers[torch]
RUN pip3 install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
RUN pip3 install -r requirements.txt

# Bundle app source
COPY . ./
# Expose
EXPOSE 6000
# Run
CMD ["python", "app.py"]

This is the stack trace that comes back from running and trying to build using this dockerfile

[+] Building 571.8s (14/17)
 => [internal] load .dockerignore                                                                                             0.0s
 => => transferring context: 34B                                                                                              0.0s
 => [internal] load build definition from Dockerfile                                                                          0.0s
 => => transferring dockerfile: 835B                                                                                          0.0s
 => [internal] load metadata for docker.io/library/ubuntu:18.04                                                               0.8s
 => [internal] load build context                                                                                             0.1s
 => => transferring context: 139.29kB                                                                                         0.1s
 => [1/13] FROM docker.io/library/ubuntu:18.04@sha256:646942475da61b4ce9cc5b3fadb42642ea90e5d0de46111458e100ff2c7031e6        0.0s
 => CACHED [2/13] RUN mkdir /usr/app                                                                                          0.0s
 => CACHED [3/13] WORKDIR /usr/app                                                                                            0.0s
 => [4/13] COPY requirements.txt ./                                                                                           0.0s
 => [5/13] RUN apt-get update                                                                                                30.8s
 => [6/13] RUN apt-get -y install python3                                                                                    15.6s
 => [7/13] RUN apt-get -y install python3-pip                                                                               214.0s
 => [8/13] RUN pip3 install virtualenv                                                                                        4.5s
 => [9/13] RUN virtualenv venv -p python3                                                                                     0.8s
 => ERROR [10/13] RUN pip3 install transformers[torch]                                                                      305.1s
------
 > [10/13] RUN pip3 install transformers[torch]:
#14 1.095 Collecting transformers[torch]
#14 1.410   Downloading https://files.pythonhosted.org/packages/2c/4e/4f1ede0fd7a36278844a277f8d53c21f88f37f3754abf76a5d6224f76d4a/
transformers-3.4.0-py3-none-any.whl (1.3MB)
#14 1.897 Collecting numpy (from transformers[torch])
#14 2.507   Downloading https://files.pythonhosted.org/packages/8f/40/ddb5109614aabad67e6fe426b3579a879b7b3cdd375eb27af467c4367ae0/
numpy-1.19.3-cp36-cp36m-manylinux1_x86_64.whl (13.4MB)
#14 5.947 Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers[torch])
#14 5.953 Collecting sentencepiece!=0.1.92 (from transformers[torch])
#14 6.174   Downloading https://files.pythonhosted.org/packages/72/e0/57edbab017a204e9f39448c1717292437a45b5f7cf3a9dbf4a9c026b03c5/
sentencepiece-0.1.94.tar.gz (507kB)
#14 6.575 Collecting sacremoses (from transformers[torch])
#14 6.714   Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/
sacremoses-0.0.43.tar.gz (883kB)
#14 7.158 Collecting requests (from transformers[torch])
#14 7.341   Downloading https://files.pythonhosted.org/packages/45/1e/0c169c6a5381e241ba7404532c16a21d86ab872c9bed8bdcd4c423954103/
requests-2.24.0-py2.py3-none-any.whl (61kB)
#14 7.404 Collecting packaging (from transformers[torch])
#14 7.551   Downloading https://files.pythonhosted.org/packages/46/19/c5ab91b1b05cfe63cccd5cfc971db9214c6dd6ced54e33c30d5af1d2bc43/
packaging-20.4-py2.py3-none-any.whl
#14 7.579 Collecting dataclasses; python_version < "3.7" (from transformers[torch])
#14 7.708   Downloading https://files.pythonhosted.org/packages/e1/d2/6f02df2616fd4016075f60157c7a0452b38d8f7938ae94343911e0fb0b09/
dataclasses-0.7-py3-none-any.whl
#14 7.725 Collecting tokenizers==0.9.2 (from transformers[torch])
#14 8.019   Downloading https://files.pythonhosted.org/packages/7c/a5/78be1a55b2ac8d6a956f0a211d372726e2b1dd2666bb537fea9b03abd62c/
tokenizers-0.9.2-cp36-cp36m-manylinux1_x86_64.whl (2.9MB)
#14 8.732 Collecting regex!=2019.12.17 (from transformers[torch])
#14 9.570   Downloading https://files.pythonhosted.org/packages/87/9f/aad666560082cb11331167cbb31cf0e8bd90af8ea4951436d1fcb2ddde44/
regex-2020.10.28-cp36-cp36m-manylinux1_x86_64.whl (666kB)
#14 9.756 Collecting protobuf (from transformers[torch])
#14 10.02   Downloading https://files.pythonhosted.org/packages/30/79/510974552cebff2ba04038544799450defe75e96ea5f1675dbf72cc8744f/
protobuf-3.13.0-cp36-cp36m-manylinux1_x86_64.whl (1.3MB)
#14 10.36 Collecting tqdm>=4.27 (from transformers[torch])
#14 10.57   Downloading https://files.pythonhosted.org/packages/93/3a/96b3dc293aa72443cf9627444c3c221a7ba34bb622e4d8bf1b5d4f2d9d08/
tqdm-4.51.0-py2.py3-none-any.whl (70kB)
#14 10.60 Collecting torch>=1.0; extra == "torch" (from transformers[torch])
#14 10.79   Downloading https://files.pythonhosted.org/packages/80/2a/58f8078744e0408619c63148f7a2e8e48cf007e4146b74d4bb67c56d161b/
torch-1.7.0-cp36-cp36m-manylinux1_x86_64.whl (776.7MB)
#14 285.6 Collecting click (from sacremoses->transformers[torch])
#14 292.4   Downloading https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/
click-7.1.2-py2.py3-none-any.whl (82kB)
#14 292.4 Collecting joblib (from sacremoses->transformers[torch])
#14 292.6   Downloading https://files.pythonhosted.org/packages/fc/c9/f58220ac44a1592f79a343caba12f6837f9e0c04c196176a3d66338e1ea8/
joblib-0.17.0-py3-none-any.whl (301kB)
#14 292.8 Requirement already satisfied: six in /usr/lib/python3/dist-packages (from sacremoses->transformers[torch])
#14 292.8 Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->transformers[torch])
#14 293.0   Downloading https://files.pythonhosted.org/packages/56/aa/4ef5aa67a9a62505db124a5cb5262332d1d4153462eb8fd89c9fa41e5d92/
urllib3-1.25.11-py2.py3-none-any.whl (127kB)
#14 293.0 Collecting chardet<4,>=3.0.2 (from requests->transformers[torch])
#14 293.2   Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/
chardet-3.0.4-py2.py3-none-any.whl (133kB)
#14 293.2 Collecting certifi>=2017.4.17 (from requests->transformers[torch])
#14 293.4   Downloading https://files.pythonhosted.org/packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/
certifi-2020.6.20-py2.py3-none-any.whl (156kB)
#14 293.4 Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests->transformers[torch])
#14 293.4 Collecting pyparsing>=2.0.2 (from packaging->transformers[torch])
#14 293.7   Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/
pyparsing-2.4.7-py2.py3-none-any.whl (67kB)
#14 293.7 Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from protobuf->transformers[torch])
#14 293.7 Collecting future (from torch>=1.0; extra == "torch"->transformers[torch])
#14 293.9   Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/
future-0.18.2.tar.gz (829kB)
#14 294.7 Collecting typing-extensions (from torch>=1.0; extra == "torch"->transformers[torch])
#14 294.9   Downloading https://files.pythonhosted.org/packages/60/7a/e881b5abb54db0e6e671ab088d079c57ce54e8a01a3ca443f561ccadb37e/
typing_extensions-3.7.4.3-py3-none-any.whl
#14 294.9 Building wheels for collected packages: sentencepiece, sacremoses, future
#14 294.9   Running setup.py bdist_wheel for sentencepiece: started
#14 295.6   Running setup.py bdist_wheel for sentencepiece: finished with status 'error'
#14 295.6   Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-o837wqyj/sent
encepiece/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __
file__, 'exec'))" bdist_wheel -d /tmp/tmprwjlzrwrpip-wheel- --python-tag cp36:
#14 295.6   /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
#14 295.6     warnings.warn(msg)
#14 295.6   running bdist_wheel
#14 295.6   running build
#14 295.6   running build_py
#14 295.6   creating build
#14 295.6   creating build/lib.linux-x86_64-3.6
#14 295.6   creating build/lib.linux-x86_64-3.6/sentencepiece
#14 295.6   copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 295.6   copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 295.6   copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 295.6   running build_ext
#14 295.6   /bin/sh: 1: pkg-config: not found
#14 295.6   ./build_bundled.sh: 8: ./build_bundled.sh: git: not found
#14 295.6   ./build_bundled.sh: 10: ./build_bundled.sh: git: not found
#14 295.6   ./build_bundled.sh: 12: cd: can't cd to sentencepiece
#14 295.6   ./build_bundled.sh: 15: ./build_bundled.sh: cmake: not found
#14 295.6   make: *** No targets specified and no makefile found.  Stop.
#14 295.6   make: *** No rule to make target 'install'.  Stop.
#14 295.6   env: 'pkg-config': No such file or directory
#14 295.6   Failed to find sentencepiece pkg-config
#14 295.6
#14 295.6   ----------------------------------------
#14 295.6   Failed building wheel for sentencepiece
#14 295.6   Running setup.py clean for sentencepiece
#14 295.8   Running setup.py bdist_wheel for sacremoses: started
#14 296.3   Running setup.py bdist_wheel for sacremoses: finished with status 'done'
#14 296.3   Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45
#14 296.4   Running setup.py bdist_wheel for future: started
#14 297.2   Running setup.py bdist_wheel for future: finished with status 'done'
#14 297.2   Stored in directory: /root/.cache/pip/wheels/8b/99/a0/81daf51dcd359a9377b110a8a886b3895921802d2fc1b2397e
#14 297.3 Successfully built sacremoses future
#14 297.3 Failed to build sentencepiece
#14 297.3 Installing collected packages: numpy, sentencepiece, click, joblib, regex, tqdm, sacremoses, urllib3, chardet, certifi, r
equests, pyparsing, packaging, dataclasses, tokenizers, protobuf, future, typing-extensions, torch, transformers
#14 301.3   Running setup.py install for sentencepiece: started
#14 301.7     Running setup.py install for sentencepiece: finished with status 'error'
#14 301.7     Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-o837wqyj/se
ntencepiece/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code,
__file__, 'exec'))" install --record /tmp/pip-7ji4iyud-record/install-record.txt --single-version-externally-managed --compile:
#14 301.7     /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
#14 301.7       warnings.warn(msg)
#14 301.7     running install
#14 301.7     running build
#14 301.7     running build_py
#14 301.7     creating build
#14 301.7     creating build/lib.linux-x86_64-3.6
#14 301.7     creating build/lib.linux-x86_64-3.6/sentencepiece
#14 301.7     copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 301.7     copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 301.7     copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 301.7     running build_ext
#14 301.7     /bin/sh: 1: pkg-config: not found
#14 301.7     mkdir: cannot create directory 'bundled': File exists
#14 301.7     ./build_bundled.sh: 8: ./build_bundled.sh: git: not found
#14 301.7     ./build_bundled.sh: 10: ./build_bundled.sh: git: not found
#14 301.7     ./build_bundled.sh: 12: cd: can't cd to sentencepiece
#14 301.7     mkdir: cannot create directory 'build': File exists
#14 301.7     ./build_bundled.sh: 15: ./build_bundled.sh: cmake: not found
#14 301.7     make: *** No targets specified and no makefile found.  Stop.
#14 301.7     make: *** No rule to make target 'install'.  Stop.
#14 301.7     env: 'pkg-config': No such file or directory
#14 301.7     Failed to find sentencepiece pkg-config
#14 301.7
#14 301.7     ----------------------------------------
#14 302.4 Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-o837wqyj/sentencepiece/setup.py';f=
getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" inst
all --record /tmp/pip-7ji4iyud-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in
 /tmp/pip-build-o837wqyj/sentencepiece/
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c pip3 install transformers[tor
ch]]: runc did not terminate sucessfully

Expected behavior

I expect transformers to be downloaded and allow me to access it from the docker container.

[thomwolf]

This will be fixed when #8073 is merged.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.