MLPClassifier drops accuracy when number of features is equal to number of nodes in the hidden layer

Odd behaviour when the number of layers is equal to the number of nodes in a single hidden layers. This may be a mathematical property of NNs or a bug, could not confirm online, please advise.

from sklearn.datasets import load_breast_cancer

import numpy as np

from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import cross_validate

df = load_breast_cancer()
X, y = load_breast_cancer().data, load_breast_cancer().target

from sklearn.model_selection import KFold

def create_CV_sets(X, y, n_splits = 5):
  skf = KFold(n_splits, shuffle=True, random_state=0)

  X_trains, y_trains, X_tests, y_tests = [], [], [], []

  for train_index, test_index in skf.split(X, y):
    X_trains.append(X[train_index,:])
    X_tests.append(X[test_index,:])
    y_trains.append(y[train_index])
    y_tests.append(y[test_index])
  
  return X_trains, y_trains, X_tests, y_tests, skf

X_trains, y_trains, X_vals, y_vals , cv = create_CV_sets(X=X, y=y, n_splits=5);

hidden_layer_sizes = [5, 10, 20, 28, 29, 30, 31, 32, 40, 50, 80, 100]

score_test_scores= {}
score_fit_times= {}
mlp_hls = []

print("Full set")
for hls in hidden_layer_sizes:
    mlp_hls.append(MLPClassifier(random_state=0, activation = "identity", hidden_layer_sizes = (hls, ), learning_rate = 'adaptive'))
    k = len(mlp_hls)-1
    scores = cross_validate(mlp_hls[k], X, y, cv=cv,return_train_score=True, scoring='accuracy')
    score_test_scores[k]=scores["test_score"].mean()
    score_fit_times[k]=scores["fit_time"].mean()
    print("Hidden Layer Sizes", hls, "\t Score: {:.3f}".format(score_test_scores[k]),"\tFit Time: {:.3f}".format(score_fit_times[k]))

[fisadev]

We did some debugging with @jbsilva in the #DataUmbrella sprint, and found out that this behavior seems to be just by chance with that particular random seed (random_state param). After trying with some other seed values, we saw that the fall in accuracy doesn’t seem to be tied to the equality between the hidden layer size and the features quantity (there are always some accuracy dips, but at random sizes). Also trying with other activation functions and datasets, yields results where the relation isn’t there anymore.

Dataset: iris
seed: 0 hidden layer sizes: (1,) test accuracy: 0.473 fit time: 0.080
seed: 0 hidden layer sizes: (2,) test accuracy: 0.007 fit time: 0.087
seed: 0 hidden layer sizes: (3,) test accuracy: 0.360 fit time: 0.096
seed: 0 hidden layer sizes: (4,) test accuracy: 0.233 fit time: 0.095 EQUAL
seed: 0 hidden layer sizes: (5,) test accuracy: 0.533 fit time: 0.098
seed: 0 hidden layer sizes: (6,) test accuracy: 0.533 fit time: 0.096
seed: 0 hidden layer sizes: (7,) test accuracy: 0.900 fit time: 0.106
seed: 0 hidden layer sizes: (8,) test accuracy: 0.587 fit time: 0.098
seed: 0 hidden layer sizes: (9,) test accuracy: 0.367 fit time: 0.097

seed: 42 hidden layer sizes: (1,) test accuracy: 0.460 fit time: 0.136
seed: 42 hidden layer sizes: (2,) test accuracy: 0.000 fit time: 0.120
seed: 42 hidden layer sizes: (3,) test accuracy: 0.807 fit time: 0.110
seed: 42 hidden layer sizes: (4,) test accuracy: 0.307 fit time: 0.119 EQUAL
seed: 42 hidden layer sizes: (5,) test accuracy: 0.547 fit time: 0.107
seed: 42 hidden layer sizes: (6,) test accuracy: 0.613 fit time: 0.105
seed: 42 hidden layer sizes: (7,) test accuracy: 0.927 fit time: 0.105
seed: 42 hidden layer sizes: (8,) test accuracy: 0.740 fit time: 0.099
seed: 42 hidden layer sizes: (9,) test accuracy: 0.700 fit time: 0.110

seed: 100 hidden layer sizes: (1,) test accuracy: 0.667 fit time: 0.150
seed: 100 hidden layer sizes: (2,) test accuracy: 0.593 fit time: 0.119
seed: 100 hidden layer sizes: (3,) test accuracy: 0.347 fit time: 0.111
seed: 100 hidden layer sizes: (4,) test accuracy: 0.373 fit time: 0.111 EQUAL
seed: 100 hidden layer sizes: (5,) test accuracy: 0.613 fit time: 0.105
seed: 100 hidden layer sizes: (6,) test accuracy: 0.333 fit time: 0.105
seed: 100 hidden layer sizes: (7,) test accuracy: 0.893 fit time: 0.110
seed: 100 hidden layer sizes: (8,) test accuracy: 0.780 fit time: 0.107
seed: 100 hidden layer sizes: (9,) test accuracy: 0.733 fit time: 0.112

seed: 999 hidden layer sizes: (1,) test accuracy: 0.213 fit time: 0.151
seed: 999 hidden layer sizes: (2,) test accuracy: 0.687 fit time: 0.122
seed: 999 hidden layer sizes: (3,) test accuracy: 0.320 fit time: 0.113
seed: 999 hidden layer sizes: (4,) test accuracy: 0.327 fit time: 0.113 EQUAL
seed: 999 hidden layer sizes: (5,) test accuracy: 0.760 fit time: 0.102
seed: 999 hidden layer sizes: (6,) test accuracy: 0.707 fit time: 0.102
seed: 999 hidden layer sizes: (7,) test accuracy: 0.527 fit time: 0.105
seed: 999 hidden layer sizes: (8,) test accuracy: 0.587 fit time: 0.106
seed: 999 hidden layer sizes: (9,) test accuracy: 0.853 fit time: 0.104

Dataset: cancer
seed: 0 hidden layer sizes: (25,) test accuracy: 0.933 fit time: 0.269
seed: 0 hidden layer sizes: (26,) test accuracy: 0.923 fit time: 0.326
seed: 0 hidden layer sizes: (27,) test accuracy: 0.909 fit time: 0.185
seed: 0 hidden layer sizes: (28,) test accuracy: 0.931 fit time: 0.340
seed: 0 hidden layer sizes: (29,) test accuracy: 0.912 fit time: 0.342
seed: 0 hidden layer sizes: (30,) test accuracy: 0.627 fit time: 0.036 EQUAL
seed: 0 hidden layer sizes: (31,) test accuracy: 0.914 fit time: 0.398
seed: 0 hidden layer sizes: (32,) test accuracy: 0.933 fit time: 0.339
seed: 0 hidden layer sizes: (33,) test accuracy: 0.237 fit time: 0.042
seed: 0 hidden layer sizes: (34,) test accuracy: 0.917 fit time: 0.365
seed: 0 hidden layer sizes: (35,) test accuracy: 0.919 fit time: 0.351

seed: 42 hidden layer sizes: (25,) test accuracy: 0.900 fit time: 0.286
seed: 42 hidden layer sizes: (26,) test accuracy: 0.907 fit time: 0.335
seed: 42 hidden layer sizes: (27,) test accuracy: 0.627 fit time: 0.038
seed: 42 hidden layer sizes: (28,) test accuracy: 0.942 fit time: 0.372
seed: 42 hidden layer sizes: (29,) test accuracy: 0.330 fit time: 0.046
seed: 42 hidden layer sizes: (30,) test accuracy: 0.910 fit time: 0.396 EQUAL
seed: 42 hidden layer sizes: (31,) test accuracy: 0.901 fit time: 0.329
seed: 42 hidden layer sizes: (32,) test accuracy: 0.589 fit time: 0.036
seed: 42 hidden layer sizes: (33,) test accuracy: 0.938 fit time: 0.364
seed: 42 hidden layer sizes: (34,) test accuracy: 0.916 fit time: 0.353
seed: 42 hidden layer sizes: (35,) test accuracy: 0.902 fit time: 0.195

seed: 100 hidden layer sizes: (25,) test accuracy: 0.916 fit time: 0.301
seed: 100 hidden layer sizes: (26,) test accuracy: 0.898 fit time: 0.329
seed: 100 hidden layer sizes: (27,) test accuracy: 0.919 fit time: 0.331
seed: 100 hidden layer sizes: (28,) test accuracy: 0.938 fit time: 0.363
seed: 100 hidden layer sizes: (29,) test accuracy: 0.896 fit time: 0.376
seed: 100 hidden layer sizes: (30,) test accuracy: 0.928 fit time: 0.339 EQUAL
seed: 100 hidden layer sizes: (31,) test accuracy: 0.921 fit time: 0.339
seed: 100 hidden layer sizes: (32,) test accuracy: 0.930 fit time: 0.326
seed: 100 hidden layer sizes: (33,) test accuracy: 0.933 fit time: 0.338
seed: 100 hidden layer sizes: (34,) test accuracy: 0.900 fit time: 0.335
seed: 100 hidden layer sizes: (35,) test accuracy: 0.156 fit time: 0.051

seed: 999 hidden layer sizes: (25,) test accuracy: 0.905 fit time: 0.280
seed: 999 hidden layer sizes: (26,) test accuracy: 0.910 fit time: 0.348
seed: 999 hidden layer sizes: (27,) test accuracy: 0.888 fit time: 0.384
seed: 999 hidden layer sizes: (28,) test accuracy: 0.916 fit time: 0.330
seed: 999 hidden layer sizes: (29,) test accuracy: 0.905 fit time: 0.348
seed: 999 hidden layer sizes: (30,) test accuracy: 0.924 fit time: 0.327 EQUAL
seed: 999 hidden layer sizes: (31,) test accuracy: 0.891 fit time: 0.344
seed: 999 hidden layer sizes: (32,) test accuracy: 0.926 fit time: 0.370
seed: 999 hidden layer sizes: (33,) test accuracy: 0.928 fit time: 0.353
seed: 999 hidden layer sizes: (34,) test accuracy: 0.924 fit time: 0.346
seed: 999 hidden layer sizes: (35,) test accuracy: 0.898 fit time: 0.315

[thomasjpfan]

Closing because https://github.com/scikit-learn/scikit-learn/issues/20083#issuecomment-869041031 shows that the performance was related to the random seed.