BUG: liblinear stucked on iris after centering the data

Describe the bug

The example on sparse Logreg surprised me because even going at extremely low regularizations (large C), only 1 feature enters the model:

Investigation lead me to check if StandardScaling X changed the graph. When X is preprocessed, the liblinear solver does not seem to converge.

Steps/Code to Reproduce

from sklearn.svm import l1_min_c
from sklearn import datasets
from sklearn import linear_model
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
import numpy as np
from time import time
print(__doc__)

# Author: Alexandre Gramfort <[e-mail]>
# License: BSD 3 clause


iris = datasets.load_iris()
X = iris.data
y = iris.target

X = X[y != 2]
y = y[y != 2]


# #############################################################################
# Demo path functions

cs = l1_min_c(X, y, loss='log') * np.logspace(0, 10, 30)


X = StandardScaler().fit_transform(X)
print("Computing regularization path ...")
start = time()
clf = linear_model.LogisticRegression(penalty='l1', solver='liblinear',
                                      tol=1e-6, max_iter=int(1e6),
                                      warm_start=True,
                                      intercept_scaling=10000.)
coefs_ = []
for c in cs:
    clf.set_params(C=c)
    clf.fit(X, y)
    coefs_.append(clf.coef_.ravel().copy())
    print(clf.coef_.ravel(), clf.intercept_)

Expected Results

the code should run fast (iris has 4 features)

Actual Results

the code gets stuck after the first regularization parameter

I guess something is happening with the scaled intercept column added because liblinear does not fit an unregularized intercept. Going to a lower intercept scaling (intercept_scaling=1) gives me the way more reasonable graph:

where I still suspect numerical errors to be responsible for the bumps when C becomes large

Versions

Happy to help if it’s a known issue @agramfort

[Rick-Mackenbach]

I’ll work on this, curious to see what is happening here.

[TomDLT]

See proposed fix in #25214

Another possibility would be to fix LIBLINEAR to avoid regularizing the intercept, backporting the fix from https://github.com/cjlin1/liblinear/commit/f68d25cc425a057cd8cdcce1554bce0172a245e8.