High cardinality categorical outputs

Howdy,

I have a dataset where the output is one of 5k categories. I also have millions of samples. The naive representation of y_indices_naive (the outputs) is:

[1,5,4300,...]

But it seems that Keras/Theano require one-hot encodings of the output.

Problem is, np_utils.to_categorical(y_indices_naive) causes an out-of-memory error because then I need a 3mil x 3k matrix.

Is there any way to get Keras to accept y_indices_naive without converting it to one-hot? I would be happy to add some code if someone would point out how to best do it.

[paulomalvar]

The trick to fix issue with the error expecting a 3D input when using sparse_categorical_crossentropy is to format outputs in a sparse 3-dimensional way. So instead of formatting the output like this:

y_indices_naive = [1,5,4300,...]

is should be formatted this way:

y_indices_naive = [[1,], [5,] , [4300,],...]

That will make Keras happy and it’ll trained the model as expected.

[fchollet]

Theano has no support for sparse operations as far as I know (and Keras certainly doesn’t either). So all data will have to be converted to dense arrays at some point. However a 5k-dimensional output space doesn’t seem very large to me.

You can solve your OOM error by one-hot encoding and training batch-by-batch instead of 3M samples at once. Break down your dataset into small batches, and for each batch:

y_batch = np_utils.to_categorical(y_indices_batch, nb_classes=5000)
model.train_on_batch(X_batch, y_batch)

As long as 1) your model fits in memory and 2) your batches are small enough, this will not cause any memory issues.