Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add initializer arguments to TransformerEncoder/TransformerDecoder

See original GitHub issue

We should add kernel_initializer and bias_initializer arguments to the TransformerEncoder/TransformerDecoder. These initializer should be passed to the dense and multi head attention layers with the encoder/decoder, and saved in the config.

Issue Analytics

State:
Created 2 years ago
Comments:8 (2 by maintainers)

Top GitHub Comments

1reaction

mattdangerwcommented, Mar 19, 2022

Thanks!

1reaction

mattdangerwcommented, Mar 18, 2022

New contributors should feel free to pick this up! Ping if more details are required.

Top Results From Across the Web

Transformer Initialization · Issue #72253 · pytorch ... - GitHub

TransformerEncoder and realized that this won't initialize parameters in a sensible way on its own. One would create an encoder like this:.

Encoder Decoder Models - Hugging Face

The EncoderDecoderModel can be used to initialize a sequence-to-sequence model ... to the specified arguments, defining the encoder and decoder configs.

Implementing the Transformer Decoder from Scratch in ...

Running this code produces an output of shape (batch size, sequence length, model dimensionality). Note that you will likely see a different ...

Models — fairseq 0.12.2 documentation

This is the legacy implementation of the transformer model that uses argparse for configuration. classmethod add_args (parser)[source]¶. Add model-specific ...

Implementing the Transformer Decoder ... - DataIntegration.info

Similar to the Transformer encoder, the Transformer decoder also ... to the random initialization of the input sequence, and the parameter ...