Illegal hardware instruction in spacy/lang/en/syntax_iterators

Running a fresh install of spacy on a FreeBSD system results in »illegal hardware instruction« and a core dump.

How to reproduce the problem

% python3.7 -m venv spacytest
% source spacytest/bin/activate
% pip --no-cache-dir install spacy
% python -m spacy download en_core_web_sm
% python -m spacy download de_core_news_sm
% python -v -c "import spacy; spacy_nlp = spacy.load('en_core_web_sm'); spacy_nlp('This is some fine text.')"
[...]
# code object from '/home/r/temp/spacytest/lib/python3.7/site-packages/spacy/lang/en/__pycache__/morph_rules.cpython-37.pyc'
import 'spacy.lang.en.morph_rules' # <_frozen_importlib_external.SourceFileLoader object at 0x8059aae90>
# /home/r/temp/spacytest/lib/python3.7/site-packages/spacy/lang/en/__pycache__/syntax_iterators.cpython-37.pyc matches /home/r/temp/spacytest/lib/python3.7/site-packages/spacy/lang/en/syntax_iterators.py
# code object from '/home/r/temp/spacytest/lib/python3.7/site-packages/spacy/lang/en/__pycache__/syntax_iterators.cpython-37.pyc'
import 'spacy.lang.en.syntax_iterators' # <_frozen_importlib_external.SourceFileLoader object at 0x8059ad150>
import 'spacy.lang.en' # <_frozen_importlib_external.SourceFileLoader object at 0x805831890>
zsh: illegal hardware instruction (core dumped)  python -v -c

Your Environment

• spaCy version: 2.2.4
• Platform: FreeBSD-12.1-RELEASE-p3-amd64-64bit-ELF
• Python version: 3.7.7

Model compatibility

The loaded model looks good:

% python -m spacy validate
✔ Loaded compatibility table

====================== Installed models (spaCy v2.2.4) ======================
ℹ spaCy installation:
/home/r/temp/spacytest/lib/python3.7/site-packages/spacy

TYPE      NAME              MODEL             VERSION
package   en-core-web-sm    en_core_web_sm    2.2.5   ✔
package   de-core-news-sm   de_core_news_sm   2.2.5   ✔

Pointers on what I might be missing or how to track down the root cause are very much appreciated.

[outpaddling]

One more thing you might try is setting BLIS_ARCH=generic before a fresh pip install. I noticed that the blis build system defaults to linux-x86_64 (even on FreeBSD-{i386,powerpc64}) and this build environment has some specific optimizations like avx2 hard-coded. If you’re running on old hardware, that could be tripping it up. I haven’t looked closely at the other packages, but I wonder if they may be doing the same. With BLIS_ARCH=generic, the basic test passed on my PowerMac G5 under FreeBSD 12.1 and an i386 build succeeded under poudriere (FreeBSD’s containerized port test / package building system).

[outpaddling]

So I added pip and wheel as dependencies for spacy and srsly. I’ll have to check setup.cfg in all the ports to make sure they’re clean. This allowed the ports spaCy build to download the models, though I have to specify a target to keep it from trying to install to /usr/local/lib/python3.7/site-packages.

Running the script below spewed a lot of what looks like debug output, but I didn’t see any errors. I’m not a spaCy user and don’t know anything about NLP, so if you want to test this further I’ll need feedback from a real user.

#!/bin/sh

for lib in en_core_web_sm de_core_news_sm; do
    if [ ! -e $HOME/spaCy/$lib ]; then
	python3.7 -m spacy download $lib --target=$HOME/spaCy
    fi
done

export PYTHONPATH=/usr/local/lib/python3.7:$HOME/spaCy
python3.7 -v -c \
    "import spacy; spacy_nlp = spacy.load('en_core_web_sm'); spacy_nlp('This is some fine text.')" \
    2>&1 | tee out