Add support for open_clip

Feature request

Add open_clip (https://github.com/mlfoundations/open_clip) support to Transformers

Motivation

open_clip has released ViT-B-32, ViT-B/16, ViT-B/16+, ViT-L/14 trained on LAION-400M and LAION-2B which are very relevant models - matching and sometimes surpassing OAI models benchmarks - but are not yet compatible with Transformers.

Also, soon a ViT-H is going to drop which will be the SOTA open source CLIP (since OAI never open sourced their ViT-H used to train DALL-E 2) - so it will also make it even more relevant to support OpenCLIP models and code

cc @rwightman

[sgugger]

Agreed with @LysandreJik . The change of activation function does not require a new model by itself (since you can set the right one in the config) but if you anticipate other modeling tweaks, a new architecture definitely makes sense.

[LysandreJik]

Hey @rwightman, excited to hear you’d like to contribute Open CLIP to transformers!

The implementation of CLIP is done using the ACT2FN activation function dictionary: https://github.com/huggingface/transformers/blob/a26c752353a127ba8e4728413806f545718a8d78/src/transformers/models/clip/modeling_clip.py#L281

If this is the only change necessary, then it should be loadable directly in the existing architecture by specifying the appopriate hidden_act configuration option.

Do you have in mind what other changes might be needed down the road for the support of additional checkpoints? I would be personally be open to having an OpenCLIP model archigtecture which could be the host for current checkpoints and upcoming checkpoints, even while unaware of the changes that might need to be done in the future (therefore with modeling code that would be a bit more dynamic than others), but I’m pinging @patrickvonplaten and @sgugger for their opinion as well.