Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

GradientBoostingClassifier: sparse data and warmstarting do not work together

See original GitHub issue

Description

Using warmstarting and sparse data for gradient boosting do not work together.

Steps/Code to Reproduce

import numpy as np
import scipy.sparse
import sklearn.datasets
import sklearn.ensemble


def get_dataset(dataset='iris', make_sparse=False):
    iris = getattr(sklearn.datasets, "load_%s" % dataset)()
    X = iris.data.astype(np.float32)
    Y = iris.target
    rs = np.random.RandomState(42)
    indices = np.arange(X.shape[0])
    train_size = int(len(indices) / 3. * 2.)
    rs.shuffle(indices)
    X = X[indices]
    Y = Y[indices]
    X_train = X[:train_size]
    Y_train = Y[:train_size]
    X_test = X[train_size:]
    Y_test = Y[train_size:]

    if make_sparse:
        X_train[:, 0] = 0
        X_train[rs.random_sample(X_train.shape) > 0.5] = 0
        X_train = scipy.sparse.csc_matrix(X_train)
        X_train.eliminate_zeros()
        X_test[:, 0] = 0
        X_test[rs.random_sample(X_test.shape) > 0.5] = 0
        X_test = scipy.sparse.csc_matrix(X_test)
        X_test.eliminate_zeros()

    return X_train, Y_train, X_test, Y_test


X_train, Y_train, _, _ = get_dataset(dataset='iris')
print(type(X_train))

classifier = sklearn.ensemble.GradientBoostingClassifier(warm_start=True)
classifier.fit(X_train, Y_train)
classifier.n_estimators += 1
classifier.fit(X_train, Y_train)
print('Fitted dense data', flush=True)

X_train, Y_train, _, _ = get_dataset(dataset='iris', make_sparse=True)
print(type(X_train))

classifier = sklearn.ensemble.GradientBoostingClassifier(warm_start=True)
classifier.fit(X_train, Y_train)
classifier.n_estimators += 1
classifier.fit(X_train, Y_train)
print('Fitted sparse data', flush=True)

Expected Results

<class 'numpy.ndarray'>
Fitted dense data
<class 'scipy.sparse.csc.csc_matrix'>
Fitted Sparse data

Actual Results

<class 'numpy.ndarray'>
Fitted dense data
<class 'scipy.sparse.csc.csc_matrix'>

Versions

Linux-4.4.0-96-generic-x86_64-with-debian-stretch-sid Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] NumPy 1.13.3 SciPy 0.19.1 Scikit-Learn 0.19.1

Update

The same issue happens for regression:

import numpy as np
import scipy.sparse
import sklearn.datasets
import sklearn.ensemble


def get_dataset(dataset='boston', make_sparse=False):
    iris = getattr(sklearn.datasets, "load_%s" % dataset)()
    X = iris.data.astype(np.float32)
    Y = iris.target
    rs = np.random.RandomState(42)
    indices = np.arange(X.shape[0])
    train_size = int(len(indices) / 3. * 2.)
    rs.shuffle(indices)
    X = X[indices]
    Y = Y[indices]
    X_train = X[:train_size]
    Y_train = Y[:train_size]
    X_test = X[train_size:]
    Y_test = Y[train_size:]

    if make_sparse:
        X_train[:, 0] = 0
        X_train[rs.random_sample(X_train.shape) > 0.5] = 0
        X_train = scipy.sparse.csc_matrix(X_train)
        X_train.eliminate_zeros()
        X_test[:, 0] = 0
        X_test[rs.random_sample(X_test.shape) > 0.5] = 0
        X_test = scipy.sparse.csc_matrix(X_test)
        X_test.eliminate_zeros()

    return X_train, Y_train, X_test, Y_test


X_train, Y_train, _, _ = get_dataset(dataset='boston')
print(type(X_train))

classifier = sklearn.ensemble.GradientBoostingRegressor(warm_start=True)
classifier.fit(X_train, Y_train)
classifier.n_estimators += 1
classifier.fit(X_train, Y_train)
print('Fitted dense data', flush=True)

X_train, Y_train, _, _ = get_dataset(dataset='boston', make_sparse=True)
print(type(X_train))

classifier = sklearn.ensemble.GradientBoostingRegressor(warm_start=True)
classifier.fit(X_train, Y_train)
classifier.n_estimators += 1
classifier.fit(X_train, Y_train)
print('Fitted sparse data', flush=True)

Issue Analytics

State:
Created 6 years ago
Comments:8 (8 by maintainers)

Top GitHub Comments

1reaction

mfeurercommented, Oct 25, 2017

@jnothman please excuse my description missing THE relevant information. Here’s the full output:

<class 'numpy.ndarray'>
Fitted dense data
<class 'scipy.sparse.csc.csc_matrix'>

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

0reactions

jnothmancommented, Oct 27, 2017

thanks.

Top Results From Across the Web

Gradient Boosting Classifier sparse matrix issue using pandas ...

I am facing a known issue with sparse matrix Conversion to dense matrix. I have applied the following solution stackoverflow but it doesnt...

sklearn.ensemble.GradientBoostingClassifier

It is a good choice for classification with probabilistic outputs. For loss 'exponential', gradient boosting recovers the AdaBoost algorithm. Deprecated since ...

Lecture 5. Ensemble Learning

Bias-variance analysis teaches us that we have two options: If model underfits (high bias, low variance): combine with other low-variance models.

Gradient Boosting Classifiers in Python with Scikit-Learn

Gradient boosting classifiers are a group of machine learning algorithms that combine many weak learning models together to create a strong ...

How Do Gradient Boosting Algorithms Handle Categorical ...

When implementations do not support categorical variables natively, ... This saves a lot of computations when the data is very sparse.