Installing spaCy with OpenMP support on OS X

I have successfully built and installed spaCy 0.100.7 with OpenMP on OS X for brew installed python. For the impatient among us, here are instructions on how to do it yourself.

Choose your adventure:

1. Install python:
   • brew install python
   • Other distributions such as Anaconda may work as well. Please let us know if you are successful!
2. (choose one):
   • llvm-3.8
   • clang-omp
   • GCC (Warning: segfauts #266)
3. Edit setup.py, then build and test spacy
4. Install spacy

Note that these instructions are for brew installed python.

Option: LLVM-3.8

1. Download the LLVM-3.8 binaries:

   • Download page: http://llvm.org/releases/download.html#3.8.0
   • Direct link for MacOS binaries: http://llvm.org/releases/3.8.0/clang+llvm-3.8.0-x86_64-apple-darwin.tar.xz

2. Install the binaries:

   For example, to install at /opt/llvm38:

   tar xJf clang+llvm-3.8.0-x86_64-apple-darwin.tar.xz
   sudo mkdir -p /opt
   sudo mv clang+llvm-3.8.0-x86_64-apple-darwin /opt/llvm38
   

3. Tell pip to use clang-3.8:

   export CC=/opt/llvm38/bin/clang
   export CXX=/opt/llvm38/bin/clang++
   export PATH=/opt/llvm38/bin:$PATH
   export C_INCLUDE_PATH=/opt/llvm38/include:$C_INCLUDE_PATH
   export CPLUS_INCLUDE_PATH=/opt/llvm38/include:$CPLUS_INCLUDE_PATH
   export LIBRARY_PATH=/opt/llvm38/lib:$LIBRARY_PATH
   export DYLD_LIBRARY_PATH=/opt/llvm38/lib:$DYLD_LIBRARY_PATH
   

Option: Clang-OMP

1. Install clang-omp Homebrew:

   brew install clang-omp
   

2. Tell pip to use clang-omp:

   export CC=clang-omp
   export CXX=clang-omp
   export PATH=/usr/local/bin:$PATH
   export C_INCLUDE_PATH=/usr/local/include/libiomp:$C_INCLUDE_PATH
   export CPLUS_INCLUDE_PATH=/usr/local/include/libiomp:$CPLUS_INCLUDE_PATH
   export LIBRARY_PATH=/usr/local/lib:$LIBRARY_PATH
   export DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH
   

Option: GCC

1. Install GCC via Homebrew:

   brew install gcc --without-multilib
   

   The --without-multilib option is required for OpenMP support.

2. Tell pip to use GCC:

   export CC=gcc-5
   export CXX=g++-5
   

WARNING: Compiling with GCC as of spaCy 0.100.5 may result in a segfault (#266).

Required: edit setup.py and install

Follow the ‘Compile from source’ instructions from spaCy documentation, with the following adjustments.

git clone https://github.com/honnibal/spaCy.git
cd spaCy
git checkout 0.100.6 # or 'master' if you wish

Edit setup.py lines 88-90 to enable OpenMP:

# if not sys.platform.startswith('darwin'):
compile_options['other'].append('-fopenmp')
link_options['other'].append('-fopenmp')

Now continue with the install instuctions as per the documentation.

virtualenv .env && source .env/bin/activate
export PYTHONPATH=`pwd`
pip install -r requirements.txt
python setup.py clean
pip install -e .
python -m spacy.en.download
pip install pytest
py.test spacy/tests/

To install spaCy outside of virtualenv and/or outside the source directory:

1. Deactivate virtualenv using the deactivate command.
2. Run pip install . in the source directory.
3. For clang/llvm, add the appropriate library path:
   • clang-omp: export DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH
   • llvm38: export DYLD_LIBRARY_PATH=/opt/llvm38/lib:$DYLD_LIBRARY_PATH

Relevant resources:

• http://docs.cython.org/src/userguide/parallelism.html
• https://clang-omp.github.io
• http://stackoverflow.com/questions/35134681/installing-openmp-on-mac-os-x-10-11
• http://stackoverflow.com/questions/27273302/cython-conditional-compile-based-on-external-value-given-via-setuptools
• http://stackoverflow.com/questions/16549893/programatically-testing-for-openmp-support-from-a-python-setup-script

Thank you for the help @honnibal @henningpeters @gushecht !

History:

• May 9 2016: Add instructions for LLVM-3.8 and use deactivate virtualenv command.
• use pip install -e . instead of python setup.py build_ext --inplace
• add instructions for installing outside of virtualenv.
• Update for 0.100.6

[gushecht]

Hey all, looks like pip install -e . was the magic formula!

For final reference:

brew install python

brew install clang-omp

git clone https://github.com/honnibal/spaCy.git

cd spaCy
git checkout 0.100.5
# Make the specified changes to setup.py

export CC=clang-omp
export CXX=clang-omp
export PATH=/usr/local/bin:$PATH
export C_INCLUDE_PATH=/usr/local/include/libiomp:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/usr/local/include/libiomp:$CPLUS_INCLUDE_PATH
export LIBRARY_PATH=/usr/local/lib:$LIBRARY_PATH
export DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH

virtualenv .env && source .env/bin/activate

export PYTHONPATH=`pwd`
pip install -r requirements.txt

python setup.py clean

pip install -e .

Thanks so much to each of you.

[sadovnychyi]

As per today clang-omp is deprecated and moved to homebrew/boneyard/clang-omp. You should use brew install llvm instead.

Environment variables I used before pip installation:

export CC=/usr/local/opt/llvm/bin/clang
export CXX=/usr/local/opt/llvm/bin/clang++
export PATH=/usr/local/opt/llvm/bin:$PATH
export C_INCLUDE_PATH=/usr/local/opt/llvm/include:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/usr/local/opt/llvm/include:$CPLUS_INCLUDE_PATH
export LIBRARY_PATH=/usr/local/opt/llvm/lib:$LIBRARY_PATH
export DYLD_LIBRARY_PATH=/usr/local/opt/llvm/lib:$DYLD_LIBRARY_PATH

Got about 2X speed up after this.