Pass coefficients to model in scikit learn

Description

I would like to train a network on a dataset and then continue training on further data, i.e., to adjust the weights of my model with the help of some data and then use these weights for further training on more data.

I thought that this would be possible with the warm_start parameter in a network model.

Steps/Code to Reproduce

from sklearn.datasets import load_boston
from sklearn import linear_model

X, y = load_boston(return_X_y=True)

X_1, X_2, y_1, y_2 = X[:len(X)/2], X[len(X)/2:], y[:len(y)/2], y[len(y)/2:]

model_1 = linear_model.ElasticNet(alpha = 0.01, warm_start=True)
in_weights = model_1.coef_
print 'Initialization : \nin_weights', in_weights
model_1.fit(X_1, y_1)
weights_1 = model_1.coef_
print '\nFirst fit:\n in_weights', in_weights, '\n \nweights_1 \n', weights_1
model_1.fit(X_2, y_2)
weights_2 = model_1.coef_
print '\n Second Fit: \nin_weights \n', in_weights, '\n \nweights_1 \n', weights_1, '\n \nweights_2 \n', weights_2

Expected Results

I expected the warm start parameter to set the weights to zero with every initialization and in case of several calls to fit use the weights of the previous fit call as starting point for the next fit if the warm_start parameter is set to True.

Actual Results

However, comparing the two options (warm_start = True/False) in practice (try in code above) yields identical results for weights and consequently for predictions. The only difference seems to be, that weights are updated retrospectively if assigned to variables in between if warm_start = True.

The following code illustrates this with a simple scikit-learn dataset and the ElasticNet as model. weights_1 changes values after the second fit such that weights_1 == weights_2 for warm_start = True (as described above), whereas weights_1 keeps the values of the first fit in case of warm_start = False (set warm_start to False in a second run for comparison).

So apparently the warm_start option is not the right approach to solve my problem. What possibilities are there then to pass coefficients, learned from a first fit to a model and continue the training with the passed coefficients as starting point?

The models I want to use this with are the MLPRegressor and the ElastcNet.

[GaelVaroquaux]

I feel that your assumptions on “warm_start” are wrong: warm_start only says that the optimization will start from the previous coefficients, but it does not say that it will end in a position controlled by these previous coefficients. Indeed, the model is fully fit on the data given, and is not influenced by the previous data.

If you want the model to be influenced by the previous data, you are looking for settings that are closer to something like on-line learning, and hence the partial_fit method of objects that support on-line learning.

[jnothman]

AFAIK, ElasticNet is only randomised if selection=‘random’. This could be better documented, perhaps.

On 2 July 2017 at 01:21, MarenMa [e-mail] wrote:

Okay, I get your results the other way round but your point is still correct and the warm_start parameter seems to work exactly as I expected indeed. Running the same code with a neural network (MLPRegressor) however, leaves me with another question: Here, for warm_start = True/False the weights are different even after the first fit. I already found out, that this can be prevented by using the random_state parameter of the MLPRegressor (see code & output).

from sklearn.datasets import load_bostonfrom sklearn.neural_network import MLPRegressor

X, y = load_boston(return_X_y=True)

half_n_samples = int(len(X)/2) X_1, X_2, y_1, y_2 = (X[:half_n_samples], X[half_n_samples:], y[:half_n_samples], y[half_n_samples:]) for random_state in [None, 3]: for warm_start in [False, True]: print’\nwarm_start:‘, warm_start, ‘\nrandom_state:’, random_state model = MLPRegressor(warm_start=warm_start, random_state=random_state) model.fit(X_1, y_1) weights_1 = model.coefs_ print’First fit coef_:\n’, model.coefs_[0][0][:10] model.fit(X_2, y_2) print’Second Fit coef_:\n’, model.coefs_[0][0][:10]

warm_start: False random_state: None First fit coef_: [-0.17946488 0.0097025 -0.08298345 -0.19441656 -0.05729989 -0.14307921 -0.04753454 -0.0268758 -0.10821514 -0.17549141] Second Fit coef_: [-0.22232356 -0.20791186 -0.21761346 -0.19104423 -0.22432604 -0.04053416 0.09225201 0.01390742 -0.22170687 0.15113488]

warm_start: True random_state: None First fit coef_: [-0.15412173 -0.19701495 -0.00825741 0.0896358 0.10279556 0.08016095 0.10262985 -0.08694756 -0.09385611 0.04355998] Second Fit coef_: [-0.15568656 -0.19628873 -0.00820137 0.09122839 0.10439327 0.08175663 0.10102941 -0.08850033 -0.09340031 0.04510905]

warm_start: False random_state: 3 First fit coef_: [ 0.04267732 0.07935861 -0.08274982 0.02425588 0.20179321 0.16434004 -0.19132201 -0.15379371 -0.22561236 -0.03907469] Second Fit coef_: [ 0.03974758 0.07971829 -0.08387953 0.02132567 0.19768738 0.16651631 -0.18878653 -0.15117453 -0.22297372 -0.04089531]

warm_start: True random_state: 3 First fit coef_: [ 0.04267732 0.07935861 -0.08274982 0.02425588 0.20179321 0.16434004 -0.19132201 -0.15379371 -0.22561236 -0.03907469] Second Fit coef_: [ 0.04068611 0.08134889 -0.08233313 0.02226467 0.19980037 0.16632802 -0.18933199 -0.15180338 -0.22362196 -0.03884355]

This appears to be reasonable in principle but I wonder why it worked for the linear model at all without setting the random_state parameter to a fixed value as the parameter equally exists but does not seem to be important for use of this model.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/scikit-learn/scikit-learn/issues/9232#issuecomment-312437928, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEz6x01OYCWxucqrye9KfPijRQ0zeCVks5sJmPngaJpZM4OGhHR .