Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Use OpenAIDiscreteVAE as pretrained model

See original GitHub issue

I notice the forward function in the OpenAIDiscreteVAE class is not implemented. I am interested in using this model as a pretrained model and train it on a medical dataset. What is the best approach to achieve this?

implement the forward function
Is it possible to initialize the parameters for DiscreteVAE class such that we can load the Openai discrete vae model in with load_state_dict( )?

Any suggestion is welcome!

Issue Analytics

State:
Created 2 years ago
Comments:6 (3 by maintainers)

Top GitHub Comments

2reactions

lucidrainscommented, Apr 27, 2021

@richcmwang there are some slight differences, so it won’t be loadable, unfortunately. one thing to note is that it is coming to light that the VQVAE from Taming Transformers is actually much better than the OpenAI one (despite the latter’s scale)

https://github.com/CompVis/taming-transformers Their repository should be more approachable for training your own VAE (or fine tuning your data from their checkpoints)

1reaction

richcmwangcommented, Apr 27, 2021

Thanks! will give it a try!

Top Results From Across the Web

Loading Google AI or OpenAI pre-trained weights ...

To load one of Google AI's, OpenAI's pre-trained models or a PyTorch saved model (an instance of BertForPreTraining saved with torch.save() ), the...

State-Of-The-Art Text to Image Generation using DALL-E

DALL-E is a novel text-to-image generation model developed by OpenAI. ... Using each pixel of a 256*256 image is impractical because it increases...

github.com-lucidrains-DALLE-pytorch_-_2021-04-19_12- ...

vae = OpenAIDiscreteVAE() # loads pretrained OpenAI VAE ... Or you can just use the official CLIP model to rank the images from...

How to Use Pretrained Models from Hugging Face ... - YouTube

In this Deep Learning Tutorial, We'll take a look at the pipeline from Hugging Face for Pretrained Neural Networks. We will go over...

Chasing fireflies transformer

The wrapper class should take care of downloading and caching the model for you ... OpenAIDiscreteVAE, DALLE vae = OpenAIDiscreteVAE() # loads pretrained...