Creating a Bias layer for use with Functional API

I am building a model using the Functional API and require a custom layer for a bias term. I have attempted to implement a custom layer called Bias which outputs a vector of length output_dim.

class Bias(Layer):
 ...
    def call(self, x=None,mask=None):
        output = self.b
        return self.activation(output)

    def get_output_shape_for(self, input_shape):
            return (1, self.output_dim)

Usage:

bias = Bias(N)()
merged = merge([X,bias],mode='concat')

The problem is that the call function expects an input tensor even if as in this case the input tensor is not used. I have attempted to produce a work around but keep getting various Keras errors.

Please can someone suggest an appropriate solution for the Bias layer.

Many thanks.

[bstriner]

If your custom layer takes an input but doesn’t actually use it in call, then it won’t be added to the compute graph.

I mention getting batch_size from another input because you will typically have at least one input to your model, maybe to another layer. Just pass that input to your bias layer as well, but don’t use it, or just use the shape from it.

Another avenue would be to absorb your bias layer into whatever the next layer is. Presumably, you are using the input at some point. You could create a larger layer, like BiasThenYaddaYadda, which takes an input, does your bias, then does whatever it does with the bias and the input.

If you’re concerned about performance, a bias layer is a bad idea. You are intentionally creating outputs of shape (batch_size, k) that are redundant. Writing the bias into your next layer is going to be the best-performing option.

I’ve dealt with the checks in topology.py and you’re better off having a slightly weird model definition than trying to fight it. That is something that would be worth enhancing but would be a long discussion.

Cheers, Ben

[bstriner]

You need an input tensor because that is how you know the batch dimension. Even if you don’t use the contents of the input tensor, you will need to use K.shape(x)[0] to get the batch dimension. You could then do K.repeat_elements to repeat the bias to be the right shape.

Let’s say your model only needs targets, not inputs, you will still pass in a dummy input np.ones(...) that is the right batch dimension.

Another possible solution is to just use a Dense layer, but set the weight matrix to 0 and remove it from trainable_weights, so W is always 0 and it only learns b.

Cheers, Ben