All Flan-T5 models configs use the incorrect activation function

System Info

The configs for all of the Flan-T5 says that the activation function is ‘gelu’ and yet ‘is_gated_act’ is set to true. This is an inherent contradiction.

Doing more digging, I realized that per Google’s original Flan-T5 checkpoints , Flan-T5 is directly instantiated from T5v1.1 LM-adapt, which all use gated-gelu

Who can help?

@younesbelkada @arthur

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

Compare the T5v1.1+LM configs to the Flan-T5 configs.

Expected behavior

“feed_forward_proj” should be “gated-gelu” and “dense_act_fn” is redundant and should be removed entirely from the config.

[LiJunnan1992]

@younesbelkada I see. Thanks so much for the explanation!

[younesbelkada]

@LiJunnan1992 No, it gets overriden by the kwargs here, check this snippet:

from transformers import T5Config

config_gated = T5Config(is_gated_act=True, hidden_act="gelu")
print(config_gated.is_gated_act)
>>> True

config_gated = T5Config(hidden_act="gelu")
print(config_gated.is_gated_act)
>>> False

config_gated = T5Config(feed_forward_proj="gated-gelu")
print(config_gated.is_gated_act)
>>> True