Add custom activation function (Swish)

TensorFlow.js version

1.1.0

Browser version

Chrome 75

Feature request

Add the Swish activation function (https://arxiv.org/abs/1710.05941). I tried implementing it by adding a custom layer, but I couldn’t get it to work. I looked into the source at activations.ts and it seems pretty easy to add Swish. Here’s my example code:

export class Swish extends Activation {
  /** @nocollapse */
  static readonly className = 'Swish';
  /**
   * Calculate the activation function.
   *
   * @param x: Input.
   * @param alpha: Scaling factor for the sigmoid function.
   * @return Output of the Swish activation.
   */
  apply(x: Tensor, alpha = 1): Tensor {
    return K.sigmoid(x.mul(alpha)).mul(x);
  }
}
serialization.registerClass(Swish);

Sadly, I don’t know the whole process of contributing (Are there any other source file to change? What about testing?), so I’d like for someone else to implement this change.

[rthadur]

@harangp thank you added to activation config

[harangp]

While the swish activation is there in the codebase, unfortunately it is not exposed, so we can’t use it in a native way (I mean, adding an activation layer, and setting the activation to ‘swish’). ActivationConfig is missing the ‘swish’ literal. Could you add it too, please? Thx.