LogisticRegression's predict_proba(X) returns (n,) instead of (n,2) for binary class

Just like sklearn’s precit_proba(X), Dask’s documentation says it returns an “array-like, shape = [n_samples, n_classes]”. However, it returns an “array-like, shape = [n_samples, 1]”, for a binary classifier. E.g. run this:

import numpy as np
from sklearn.linear_model import LogisticRegression
from dask_ml.linear_model import LogisticRegression as DaskLogit

# Data
X = np.random.rand(200,20)
y = np.random.rand(200).round(0) # random binary labels

# Dask 
lrdask = DaskLogit()
lrdask.fit(X, y)
print("Logit Reg 'predict_proba' shape Dask\t", lrdask.predict_proba(X).shape)

# SKLearn
lrsklr = LogisticRegression()
lrsklr.fit(X, y)
print("Logit Reg 'predict_proba' shape SKLearn\t", lrsklr.predict_proba(X).shape)

Of course, no information is lost, but this does break stuff when used in other code. E.g. in GridSearchCV’s scorer I get:

C:\ProgramData\Anaconda3\lib\site-packages\sklearn\metrics\scorer.py in __call__(self, clf, X, y, sample_weight)
    184 
    185                 if y_type == "binary":
--> 186                     y_pred = y_pred[:, 1]
    187                 elif isinstance(y_pred, list):
    188                     y_pred = np.vstack([p[:, -1] for p in y_pred]).T

IndexError: too many indices for array

predict_proba(X)'s return value is indeed implemented differently in sklearn and dask-ml: sklearn:

def _predict_proba_lr(self, X):
	...
	return np.vstack([1 - prob, prob]).T

dask-ml:

def predict_proba(self, X):
	...
	return sigmoid(dot(X_, self._coef))

Is this intentional and would changing this break code that currently expects Dask to return (n,) and not (n,2)? Or is this an oversight, and should it return (n,2) as per documentation?

[TomAugspurger]

Still open, if you’re interested in working on it.

IIUC, there are two issues:

1. For binary logicist regression, properly (N,) shape output to (N, 2).
2. Implement multinomial logistic regression.

The first sounds relatively easier, so we might want to start with that. That would be around https://github.com/dask/dask-ml/blob/master/dask_ml/linear_model/glm.py#L248

[TomAugspurger]

Nope, nothing beyond we should match scikit-learn.