rank_strings BrokenPipeError: [Errno 32] Broken pipe on macOS

macOS 10.14.6 python 3.7.6 homebrew

• https://github.com/fireeye/stringsifter#running-from-the-command-line

Traceback (most recent call last):
  File "/usr/local/bin/rank_strings", line 8, in <module>
    sys.exit(argmain())
  File "/usr/local/lib/python3.7/site-packages/stringsifter/rank_strings.py", line 138, in argmain
    args.scores, args.batch)
  File "/usr/local/lib/python3.7/site-packages/stringsifter/rank_strings.py", line 27, in main
    ranker = joblib.load(os.path.join(modeldir, "ranker.pkl"))
  File "/usr/local/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 598, in load
    obj = _unpickle(fobj, filename, mmap_mode)
  File "/usr/local/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 526, in _unpickle
    obj = unpickler.load()
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 1088, in load
    dispatch[key[0]](self)
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 1376, in load_global
    klass = self.find_class(module, name)
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 1426, in find_class
    __import__(module, level=0)
  File "/usr/local/lib/python3.7/site-packages/lightgbm/__init__.py", line 8, in <module>
    from .basic import Booster, Dataset
  File "/usr/local/lib/python3.7/site-packages/lightgbm/basic.py", line 33, in <module>
    _LIB = _load_lib()
  File "/usr/local/lib/python3.7/site-packages/lightgbm/basic.py", line 28, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 442, in LoadLibrary
    return self._dlltype(name)
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/usr/local/lib/python3.7/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/gcc/lib/gcc/8/libgomp.1.dylib
  Referenced from: /usr/local/lib/python3.7/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found
Traceback (most recent call last):
  File "/usr/local/bin/flarestrings", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/stringsifter/flarestrings.py", line 29, in main
    print(match.group().decode('ascii'))
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

Requirement already satisfied, skipping upgrade: lightgbm==2.1.2 in /usr/local/lib/python3.7/site-packages (from stringsifter) (2.1.2)
Requirement already satisfied, skipping upgrade: numpy==1.17.1 in /usr/local/lib/python3.7/site-packages (from stringsifter) (1.17.1)
Requirement already satisfied, skipping upgrade: scikit-learn==0.21.3 in /usr/local/lib/python3.7/site-packages (from stringsifter) (0.21.3)
Requirement already satisfied, skipping upgrade: joblib==0.13.2 in /usr/local/lib/python3.7/site-packages (from stringsifter) (0.13.2)
Requirement already satisfied, skipping upgrade: pytest==3.10.1 in /usr/local/lib/python3.7/site-packages (from stringsifter) (3.10.1)
Requirement already satisfied, skipping upgrade: fasttext==0.9.1 in /usr/local/lib/python3.7/site-packages (from stringsifter) (0.9.1)
Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.7/site-packages (from lightgbm==2.1.2->stringsifter) (1.3.1)
Requirement already satisfied, skipping upgrade: py>=1.5.0 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (1.8.0)
Requirement already satisfied, skipping upgrade: more-itertools>=4.0.0 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (7.2.0)
Requirement already satisfied, skipping upgrade: six>=1.10.0 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (1.12.0)
Requirement already satisfied, skipping upgrade: setuptools in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (42.0.2)
Requirement already satisfied, skipping upgrade: atomicwrites>=1.0 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (1.3.0)
Requirement already satisfied, skipping upgrade: pluggy>=0.7 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (0.12.0)
Requirement already satisfied, skipping upgrade: attrs>=17.4.0 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (18.2.0)
Requirement already satisfied, skipping upgrade: pybind11>=2.2 in /usr/local/lib/python3.7/site-packages (from fasttext==0.9.1->stringsifter) (2.3.0)
Requirement already satisfied, skipping upgrade: importlib-metadata>=0.12 in /usr/local/lib/python3.7/site-packages (from pluggy>=0.7->pytest==3.10.1->stringsifter) (0.20)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in /usr/local/lib/python3.7/site-packages (from importlib-metadata>=0.12->pluggy>=0.7->pytest==3.10.1->stringsifter) (0.6.0)
Building wheels for collected packages: stringsifter
  Building wheel for stringsifter (setup.py) ... done
  Created wheel for stringsifter: filename=stringsifter-0.20191202-py3-none-any.whl size=1932401 sha256=9e0f0c617e547b96f9ef88995e6d5006ae9aa54a8bc16cc16b077f1284f7e832
  Stored in directory: /Users/admin/Library/Caches/pip/wheels/3c/89/aa/c3bc7a6c171c52f5d4bea07724f5f2fd9fc4e7ad23b5946a8c
Successfully built stringsifter
Installing collected packages: stringsifter
  Attempting uninstall: stringsifter
    Found existing installation: stringsifter 0.20190907
    Uninstalling stringsifter-0.20190907:
      Successfully uninstalled stringsifter-0.20190907
Successfully installed stringsifter-0.20191202

probably same issue #10 is reporting, but I can’t tell for sure because there is limited info @phtully ?

[gabemarshall]

In case it helps anyone, what worked for me was brew install lightgbm (which also installs libomp) then pip uninstall lightgbm && pip install -U lightgbm

[wesinator]

Sorry for the delayed reply (was on vacation), but thanks for using StringSifter, @wesinator! Your stack trace seems to indicate that lightgbm might not be installed properly, or one of it’s dependencies is missing. I’d like to help you fix this and change the docs/code to make sure other users can troubleshoot the issue in the future.

Let’s start with this, if you open up a python interpreter within that environment, and try to import lightgbm manually, do you get the same OsError or BrokenPipeError in your stack trace?

yes, import lightgbm gave the following error:

OSError: dlopen(/usr/local/lib/python3.7/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/gcc/lib/gcc/8/libgomp.1.dylib
  Referenced from: /usr/local/lib/python3.7/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found

running the following fixed it:

brew install libomp
pip install -U lightgbm

ERROR: stringsifter 0.20191202 has requirement lightgbm==2.1.2, but you'll have lightgbm 2.3.1 which is incompatible.
Installing collected packages: lightgbm
  Attempting uninstall: lightgbm
    Found existing installation: lightgbm 2.1.2
    Uninstalling lightgbm-2.1.2:
      Successfully uninstalled lightgbm-2.1.2
Successfully installed lightgbm-2.3.1

works with latest lightgbm despite stringsifter req version