Types for functions activations used in Sequential?

Is there interest in a pull request adding a few model layers representing activations for use in Sequential models? Many Pytorch examples use sequential, and this could be helpful for beginners trying to translate pytorch models to diffsharp.

I recognize that you may not want this because it increases the maintenance burden and --> covers many Sequential use cases. Hence the question.

For example, the pytorch.org quickstart model definition example uses sequential, and I believe this is the correct translation:

type ReLU() =
    inherit Model()
    override _.ToString() = sprintf "ReLU()"
    override m.forward(value) = 
        dsharp.relu(value)

type Flatten(?startDim:int,?endDim:int) =
    inherit Model()
    let startDim = defaultArg startDim 1
    override _.ToString() = sprintf "Flatten()"
    override m.forward(value) =
        dsharp.flatten(value, startDim, ?endDim=endDim) 
        
type NeuralNetwork() =
    inherit Model()
    let linear_relu_stack = Sequential([
        Linear(28*28, 512)
        ReLU()
        Linear(512, 512)
        ReLU()
        Linear(512, 10)
    ])
    do base.addModel([linear_relu_stack])
    member self.flatten = Flatten()
    
    override self.forward(x) =
        let x = self.flatten.forward(x)
        let logits = linear_relu_stack.forward(x)
        logits


let model = NeuralNetwork()

[nhirschey]

Thank you for the thoughtful response. I am not aware of Torch’s history, so that is helpful.

I prefer model3 with --> syntax, but given how frequently I have seen Sequential in PyTorch examples, I think that your function-to-model wrapper is a worthy addition. It is an elegant solution.

This context may help explain my motivations. I want to be able to tell my students:

“Here’s the PyTorch.org quickstart in Python. Here’s the literal line by line translation using F# and Diffsharp. See, it’s almost exactly the same. Now, look at the same model in DiffSharp --> style, isn’t that simpler?”

I think there’s value in having the ability to do a line-by-line translation of whatever PyTorch.org has as the quickstart, even if it is more common for people to use --> once they’re familiar with the library.

[nhirschey]

Thank you for adding Model(...). This looks fantastic and the two versions (model 4 and let ReLu() = Model(dsharp.relu)) are sufficient to cover my use case.

I’ll close this issue as addressed. With the addition of Model(...), I no longer feel the need to submit a PR adding Model objects for activation functions.

Thanks again for this great library.