Does LambdaLayer need BatchNorm and activation after it?

Hello, I’m trying to reproduce of this. I’m build the LambdaResnet,a little question is BatchNorm and activation are needed after this(LambdaLayer)? Thanks.

[PistonY]

Hi @lucidrains, thanks for reply. I tested them all, I think you’re right. When apply bn+relu the val accuracy doesn’t grow. This is my final implement.

Now I’m training LambdaResnet50,it’s looking good: I use the standard training step in my project same with Resnet50 except batch_size set to 64.

Epoch 28 result: Train Acc:0.44 Loss:2.54 Val Acc:0.48 Loss:3.1748e+08 Some observations are:

1. parameters and GFLOPs are small but training speed and gpu memory cost are still high.
2. Mix-Precision(FP16) training by torch.cuda.amp would make train loss nan, that’s make me could only train with 64 batch_size.
3. Val loss is strange.
4. Convergence is much slower than Resnet50.

[PistonY]

@lucidrains Unfortunately, I only got 76.1 best top1 on val set(79.2 on train set). I’d better wait author release their code.