How do we know which backend functions are differentiable / not?

Hi, I’m trying to define custom loss functions for my model but have come across this error several times during development:

ValueError: An operation has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.

I understand the need for differentiable loss functions, but I was wondering if there is any documentation on which functions in Keras are differentiable and which functions are not. My first instinct was that any of the Keras backend functions are differentiable and hence usable in my loss functions, but clearly (as seen in the error message) that is not the case.

I feel like it would be very helpful to have a list I can refer to, instead of discovering by trial and error as I have been doing so far. Does such a resource already exist, and if not, can we make one?

[JunShern]

Hi @KrishnanParameswaran , I couldn’t figure out a perfect way to do it but I was able to get what I needed done through trial and error, using a simple example like the following:

# Loss function
def custom_loss(input_x, output_x):
    loss = output_x - input_x
    return loss

# Dummy model
input_layer = Input(shape=INPUT_SHAPE)
hidden_layer = Dense(10)(input_layer)
output_layer = Reshape(OUTPUT_SHAPE)(hidden_layer)

model = Model(input_layer, output_layer)
model.compile(optimizer='sgd', loss=test_loss)

# Train the model
model.fit(train_x, train_y)

Using this sort of trial and error I did take note of which ops I found to be usable and which were problematic:

Differentiable ops

• Indexing
• +, -, *, /
• K.squeeze
• K.spatial_2d_padding

Non-differentiable ops

• K.argmax
• K.round
• K.eval
• K.greater
• K.cast bool to float, int to float

Also note that it’s not just the op itself which determines differentiability. It’s about how you chain the ops. For example, K.mean(y_true) has no gradient and will error. On the other hand, K.mean(y_true - y_pred) does have a gradient and can train without issue.

But once you have a simple example to test on, it’s actually not too difficult to get the loss you want. I managed to work out a number of custom losses by reformulating my objectives using common arithmetic operations which can actually be quite descriptive when used creatively.

[roya90]

Hey, Is K.cast for float64 to float32 differentiable?