Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

FusedLayerNorm vs torch.nn.LayerNorm

See original GitHub issue

What’s the advantage of using the FusedLayerNorm over torch.nn.LayerNorm? I’m running into an issue with using TorchScript and I’m wondering if I can replace the former with the latter.

The deeper question is: Is the apex version of layer norm significantly optimized over the standard pytorch version or is it simply a legacy of when pytorch did not have a built in layer norm function?

Issue Analytics

State:
Created 4 years ago
Reactions:1
Comments:13 (1 by maintainers)

Top GitHub Comments

5reactions

myleottcommented, May 1, 2020

I observe the same issue as @ngoyal2707 on PyTorch 1.5 – torch.nn.LayerNorm is slower than apex.FusedLayerNorm for shapes typical in NLP models. For example: (512, 16, 1024) with normalization over the last dimension is slower using torch.nn.LayerNorm.

0reactions

pommedeterresauteecommented, Jul 16, 2020

@hitvoice 2080 TI and apex from master branch at the time of my precedent message, so June 16th

Top Results From Across the Web

LayerNorm — PyTorch 1.13 documentation

This layer uses statistics computed from input data in both training and evaluation modes. Parameters: normalized_shape (int or list or torch.Size) –. input ......

Fused LayerNorm - Composer

Fused LayerNorm is implemented by performing model surgery, which looks for instances of torch.nn.LayerNorm and replaces them with a apex.normalization.

apex.normalization.fused_layer_norm - GitHub Pages

This layer uses statistics computed from input data in both training and evaluation modes. Parameters. normalized_shape (int or list or torch.Size) –. input ......

Layer Normalization in Pytorch (With Examples) - Wandb

To do so, you can use torch.nn.LayerNorm(). For convolutional neural networks however, one also needs to calculate ...

Source code for fairseq.modules.layer_norm

... source tree. import torch import torch.nn as nn import torch.nn.functional as F try: from apex.normalization import FusedLayerNorm as _FusedLayerNorm ...