Image GPT

🌟 New model addition

Model description

OpenAI just announced Image GPT: https://openai.com/blog/image-gpt/

Although image rendering would be out of scope for Transformers, the RGB generation would still be in scope and it would be best to port the weights to a GPT2LMModel.

However, it’s not immediately clear here how the tokenization is implemented in the downloaded model. (no separate vocab.json)

Open source status

• the model implementation is available: https://github.com/openai/image-gpt
• the model weights are available: see README above
• who are the authors: @openai

[apeguero1]

Hey @minimaxir! Here’s a colab which loads the weights into a subclass of GPT2LMHeadModel and demonstrates unconditional image generation and conditional image completion.

Some differences I’ve found between Image-GPT and GPT2 which are reflected in the subclass.

1. Image-GPT layer normalization doesn’t subtract off the mean
2. different activations used in the MLP
3. In Image-GPT, the input and output embeddings are not tied
4. Image-GPT has an extra learned “sos” token embedding which is concatenated at the beginning of the sequence
5. The GPT2 [n_embd, 3*n_embd] dimensional linear layer, c_attn, which produces queries, keys, and values is instead split into 3 separate linear layers each with dimension [n_head, n_embd/n_head, n_embd] in Image-GPT (this only affects how to load the weights and not the actual model).
6. In Image-GPT, the conv1d module doesn’t have a bias term

So what’s our next step to add this to the repo?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.