[QUESTION] Support for ortools 9.4 solvers natively on MacOS with Silicon?

NOTE: This issue sounds similar to https://github.com/guillermo-navas-palencia/optbinning/issues/185 and I’ve opened https://github.com/google/or-tools/issues/3485 to get advice from that angle. It’s unclear to me if this is a bug, an incompatibility that should be addressed/documented, or simply a usage mistake.

Problem Description

My goal is to use optbinning with the latest ortools (9.4.1874) on a MacOS Monterey machine with Apple Silicon and an arm64 python platform.machine() architecture (no Rosetta2). I’m currently trying to use optbinning==0.15.0.

I am using ContinuousOptimalBinning.fit(), but when it eventually runs the optimizer.solve() method, the code seems to silently fail shortly thereafter.

NOTE: Here is a minimal reproduction notebook file and titanic dataset. optimalBinnerRepro.zip

Minimal Reproduction Example

# %%
import pandas as pd
from optbinning import OptimalBinning, ContinuousOptimalBinning

# %%
df = pd.read_csv('./titanic.csv')
df.shape

# %%
missing_cols = df.columns[[df[col].isna().any() for col in df]].tolist()
missing_cols

# %%
dropped_idx = df['Age'][df['Age'].isnull()].index.tolist()
df = df.drop(dropped_idx, axis=0)
df.shape

# %%
# Initialize binner

continuousBinner = ContinuousOptimalBinning(max_n_bins=20, min_bin_size=0.05, split_digits=2)

# %%
# Continuous 

y = df['Fare'].values
X = df['Age']
# The following line will result in a process crash.
age_binned_multiclass = continuousBinner.fit(X, y)
continuousBinner.splits

Once inside of ortools code, execution seems to stop here: https://github.com/google/or-tools/blob/82750ac12f1ee5354e1c7869894d9af3508778f2/ortools/sat/python/cp_model.py#L2190

My debugger is unable to catch any traceback or error from this series of invocations, so I don’t really know what’s going wrong. The trail goes cold here after invoking this method:

https://github.com/google/or-tools/blob/82750ac12f1ee5354e1c7869894d9af3508778f2/ortools/sat/python/swig_helper.cc#L60

---

I suspected an architecture incompatibility issue, but this version of ortools has a pre-built wheel for arm64, so I was inclined to trust it.

• Does optbinning work with ortools==9.4.1874 on macos arm64? Is there any version combination where this would work?
• Are there solver incompatibilities at play here?
• Any advice on how to better debug this situation?

[tboddyspargo]

UPDATE: It works after clearing my venv and pip cache and installing only the packages necessary for the minimal repro I provided.

The minimal repro example also helped surface an exception from the crashing thread. When run with my original dependency lockfile from a standard python interpreter (not a jupyter notebook), I see this line get printed:

Bus error: 10

output of `pip freeze` in working example

absl-py==1.2.0
contourpy==1.0.5
cvxpy==1.2.1
cycler==0.11.0
ecos==2.0.10
fonttools==4.37.4
joblib==1.2.0
kiwisolver==1.4.4
matplotlib==3.6.0
numpy==1.23.3
optbinning==0.15.1
ortools==9.4.1874
osqp==0.6.2.post5
packaging==21.3
pandas==1.5.0
Pillow==9.2.0
protobuf==4.21.7
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.4
qdldl==0.1.5.post2
ropwr==0.3.0
scikit-learn==1.1.2
scipy==1.9.1
scs==3.2.0
six==1.16.0
threadpoolctl==3.1.0

output of `pip freeze` in broken example

absl-py==1.2.0
aiorwlock==1.1.0
alembic==1.7.7
anyio==3.6.1
appdirs==1.4.4
appdirs-stubs==0.1.0
appnope==0.1.3
APScheduler==3.9.1
apsw==3.38.5.post1
asn1crypto==1.5.1
astroid==2.9.3
asttokens==2.0.5
async-generator==1.10
async-property==0.2.1
async-timeout==4.0.2
attrs==21.4.0
aws-sam-translator==1.46.0
azure-core==1.22.1
azure-storage-blob==12.10.0
backcall==0.2.0
backports-datetime-fromisoformat==1.0.0
beautifulsoup4==4.10.0
black==22.1.0
blis==0.7.7
boto3==1.21.24
botocore==1.24.24
bs4==0.0.1
build==0.8.0
cachetools==5.0.0
catalogue==2.0.7
catboost==1.0.6
certifi==2021.10.8
cffi==1.15.0
cfn-lint==0.61.1
charset-normalizer==2.0.12
click==8.0.4
clickhouse-driver==0.2.0
cloudpickle==2.0.0
cmdstanpy==1.0.2
coloredlogs==15.0.1
colormath==3.0.0
contextlib2==21.6.0
contourpy==1.0.5
convertdate==2.4.0
cryptography==36.0.2
cvxpy==1.2.1
cx-Oracle==8.3.0
cycler==0.11.0
cymem==2.0.6
Cython==0.29.28
databricks-sql-connector==2.0.2
db-dtypes==0.3.1
debugpy==1.5.1
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.13
dill==0.3.4
dnspython==2.2.1
docker==5.0.3
ecos==2.0.10
editdistance==0.6.0
elasticsearch==7.13.4
elasticsearch-dbapi==0.2.9
ephem==4.1.3
et-xmlfile==1.1.0
executing==0.8.3
Faker==13.3.2
fastdiff==0.3.0
fasttreeshap==0.1.1
firebolt-sdk==0.8.1
firebolt-sqlalchemy==0.5.1
Flask==1.0.2
Flask-Migrate==2.3.1
Flask-Script==2.0.5
Flask-SQLAlchemy==2.5.1
flattentool==0.15.4
fonttools==4.37.4
future==0.18.2
gitdb==4.0.9
GitPython==3.1.27
google-api-core==1.33.1
google-api-python-client==2.42.0
google-auth==2.6.2
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.5.1
google-cloud-bigquery==2.34.2
google-cloud-bigquery-storage==2.13.0
google-cloud-core==2.2.3
google-cloud-storage==2.2.1
google-crc32c==1.3.0
google-resumable-media==2.3.2
googleapis-common-protos==1.56.4
graphviz==0.19.1
grpcio==1.44.0
grpcio-status==1.44.0
h11==0.12.0
h2==4.1.0
hijri-converter==2.2.3
holidays==0.13
hpack==4.0.0
html-sanitizer==1.9.3
httpcore==0.15.0
httplib2==0.20.4
httpx==0.23.0
humanfriendly==10.0
hyperframe==6.0.1
hyperopt==0.2.7
idna==3.3
ijson==3.1.4
imbalanced-learn==0.9.1
importlib-metadata==3.7.3
importlib-resources==5.4.0
iniconfig==1.1.1
ipython==8.1.1
isodate==0.6.1
isort==5.10.1
itsdangerous==2.0.1
jedi==0.18.1
Jinja2==3.0.3
jmespath==1.0.0
joblib==1.2.0
JPype1==1.4.0
jschema-to-python==1.2.3
jsonpatch==1.32
jsonpickle==2.2.0
jsonpointer==2.3
jsonref==0.2
jsonschema==3.2.0
junit-xml==1.9
kiwisolver==1.4.4
korean-lunar-calendar==0.2.1
kubernetes==23.3.0
kylinpy==2.8.4
langcodes==3.3.0
lazy-object-proxy==1.7.1
lightgbm==3.3.2
llvmlite==0.39.0
LunarCalendar==0.0.9
lxml==4.8.0
Mako==1.2.0
MarkupSafe==2.1.1
matplotlib==3.6.0
matplotlib-inline==0.1.3
mccabe==0.6.1
mirakuru==2.4.2
msrest==0.6.21
murmurhash==1.0.8
mypy-extensions==0.4.3
mysql-connector-python==8.0.29
networkx==2.7.1
numba==0.56.2
numpy==1.23.3
oauth2client==4.1.3
oauthlib==3.2.0
odfpy==1.4.1
openpyxl==3.0.9
optbinning==0.15.1
ortools==9.4.1874
oscrypto==1.3.0
osqp==0.6.2.post5
outcome==1.1.0
packaging==21.3
pamqp==2.3.0
pandas==1.5.0
pandas-gbq==0.17.4
parso==0.8.3
pathspec==0.9.0
pathy==0.6.1
patsy==0.5.2
pbr==5.9.0
Pebble==4.6.3
pep517==0.13.0
pexpect==4.8.0
phonenumbers==8.12.45
pickleshare==0.7.5
pika==1.2.0
Pillow==9.0.1
pinotdb==0.3.11
pip-tools==6.9.0
platformdirs==2.5.1
plotly==5.6.0
pluggy==1.0.0
plumbum==1.7.2
port-for==0.6.2
preshed==3.0.6
prometheus-client==0.13.1
prompt-toolkit==3.0.28
prophet==1.1
proto-plus==1.20.3
protobuf==3.20.2
psutil==5.9.1
psycopg2-binary==2.9.3
ptyprocess==0.7.0
pure-eval==0.2.2
pure-sasl==0.6.2
py==1.11.0
py4j==0.10.9.5
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyathena==2.5.1
pyathenajdbc==3.0.1
pybind11==2.10.0
pycparser==2.21
pycryptodome==3.14.1
pycryptodomex==3.14.1
pydantic==1.8.2
pydata-google-auth==1.4.0
pydocstyle==6.1.1
pydot==1.4.2
pydotplus==2.0.2
PyDrive==1.3.1
pydruid==0.6.3
Pygments==2.11.2
PyHive==0.6.5
PyJWT==2.3.0
pylint==2.12.2
PyMeeus==0.5.11
pymongo==4.0.2
PyMySQL==1.0.2
PyNaCl==1.5.0
pynndescent==0.5.7
pyodbc==4.0.32
pyOpenSSL==21.0.0
pyparsing==2.4.7
pyrsistent==0.18.1
PySocks==1.7.1
pytest==7.1.1
pytest-env==0.6.2
pytest-mock==3.7.0
pytest-rabbitmq==2.2.1
pytest-timeout==2.1.0
python-dateutil==2.8.2
python-dotenv==0.20.0
python-json-logger==2.0.2
pytz==2022.4
PyYAML==6.0
qdldl==0.1.5.post2
rabbitpy==2.0.1
readerwriterlock==1.0.9
redis==4.2.0
requests==2.27.1
requests-mock==1.9.3
requests-oauthlib==1.3.1
rfc3986==1.5.0
ropwr==0.3.0
rpyc==5.1.0
rsa==4.8
s3transfer==0.5.2
salesforce-merlion==1.1.2
sarif-om==1.0.4
sasl==0.3.1
schema==0.7.5
scikit-learn==1.1.2
scipy==1.9.1
scs==3.2.0
selenium==4.1.3
setuptools-git==1.2
shap==0.40.0
shillelagh==1.0.14
simplenlg==0.2.0
six==1.16.0
slicer==0.0.7
smart-open==5.2.1
smmap==5.0.0
snapshottest==0.6.0
sniffio==1.2.0
snowballstemmer==2.2.0
snowflake-connector-python==2.7.6
snowflake-sqlalchemy==1.3.3
sortedcontainers==2.4.0
soupsieve==2.3.1
spacy==3.3.1
spacy-legacy==3.0.9
spacy-loggers==1.0.1
spectra==0.0.11
SQLAlchemy==1.4.27
sqlalchemy-bigquery==1.4.3
sqlalchemy-cockroachdb==1.4.3
sqlalchemy-databricks==0.2.0
sqlalchemy-dremio==1.2.1
sqlalchemy-drill==1.1.2
sqlalchemy-fdw==0.3.0
sqlalchemy-hana==0.5.0
sqlalchemy-redshift==0.8.9
sqlalchemy-solr==0.2.2.1
sqlalchemy-trino==0.5.0
sqlalchemy-vertica-python==0.5.10
sqlalchemy-views==0.3.1
sqlparse==0.4.2
SRL==0.1.0
srsly==2.4.4
stack-data==0.2.0
statsforecast==0.5.4
statsmodels==0.13.2
tenacity==8.0.1
teradatasql==17.10.0.9
teradatasqlalchemy==17.0.0.2
termcolor==1.1.0
textdistance==4.2.2
thinc==8.0.15
thinc-apple-ops==0.0.7
threadpoolctl==3.1.0
thrift==0.15.0
thrift-sasl==0.4.3
toml==0.10.2
tomli==2.0.1
tqdm==4.63.0
traitlets==5.1.1
trino==0.313.0
trio==0.20.0
trio-websocket==0.9.2
twilio==7.7.1
typer==0.4.2
typing_extensions==4.1.1
tzlocal==2.1
ujson==5.1.0
umap-learn==0.5.3
uritemplate==4.1.1
urllib3==1.26.9
vertica-python==1.1.0
wasabi==0.10.1
wasmer==1.1.0
wasmer-compiler-cranelift==1.1.0
wcwidth==0.2.5
websocket-client==1.3.1
Werkzeug==2.0.3
wordcloud==1.8.1
wrapt==1.13.3
wsproto==1.1.0
xgboost==1.6.0
xlwt==1.3.0
xmltodict==0.12.0
zipp==3.7.0

I was able to further reduce the package version differences between those two examples so the issue doesn’t seem to be related to the versions of any of the dependencies. That leaves the possibility that the issue is related to the presence of one of the many “extra” packages in my original lockfile which are absent in the working minimal repro requirements. One other difference is that I currently have a work-around in place to allow pyodbc to be built form source on macos arm64.

I’m going to try to continue eliminating variables to see where that gets me.

[tboddyspargo]

Thank you so much for your response, @guillermo-navas-palencia! I really appreciate the help investigating this.

• Could you try the OptimalBinning class using solver="mip" (ContinuousOptimalBinning always uses CP-SAT solver)?

I can confirm that solver="mip" did not result in a process crash.

Using macos-12 and ortools 9.4.1874 all tests passed.

That’s really great to hear!

That makes it sound like this is more likely related to a problem on my end… I guess the first thing I’ll try is to clear my pip cache and start a fresh virtual environment and try again, focusing on my minimal repro example.