Windows and Sentence Piece

🐛 Bug

Describe the bug

It appears one of the requirements sentencepiece is not built for windows machines.

This causes an import error:

     1 import torch
----> 2 import torchtext
      3 import numpy as np
      4 import pandas as pd

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-dev\lib\site-packages\torchtext\__init__.py in <module>
----> 1 from . import data
      2 from . import datasets
      3 from . import utils
      4 from . import vocab
      5 from . import experimental

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-dev\lib\site-packages\torchtext\data\__init__.py in <module>
      8 from .pipeline import Pipeline
      9 from .utils import get_tokenizer, interleave_keys
---> 10 from .functional import generate_sp_model, \
     11     load_sp_model, \
     12     sentencepiece_numericalizer, \

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-dev\lib\site-packages\torchtext\data\functional.py in <module>
----> 1 import sentencepiece as spm
      2 import re
      3 
      4 __all__ = [
      5     "generate_sp_model", "load_sp_model",

ModuleNotFoundError: No module named 'sentencepiece'

Expected behavior module imports without issue.

Environment

Please copy and paste the output from our environment collection script (or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
python -c "import torchtext; print(\"torchtext version is \", torchtext.__version__)"

• PyTorch Version (e.g., 1.0): latest
• OS (e.g., Linux): Windows
• How you installed PyTorch (conda, pip, source): conda
• Build command you used (if compiling from source):N/A
• Python version:3.6
• CUDA/cuDNN version:
• GPU models and configuration:
• Any other relevant information: Install torchtext from conda.

[zhangguanheng66]

@achapkowski @zhangguanheng66 Is this still a Python-only package? If yes, then you could compile it as a noarch package. See here for more details.

Yes. It’s currently python-only package but we may expand it with some cpp soon. And a noarch package is what we are doing now.

[peterjc123]

It seems the sentencepiece packages are hosted on PyPI. https://pypi.org/project/sentencepiece/0.1.85/#files