Recent commit results in higher VRAM usage

The commit in question is ed91ab4a30ffdcf7e8773e6b434816f79f5fead8. My system has 8GB of VRAM and is using Torch with ROCm support.

Here are my results with that commit and another from the one before it:

$ python image_from_text.py --text="alien life" --mega --torch --seed 100
Namespace(mega=True, torch=True, text='alien life', seed=100, image_path='generated', sample_token_count=256)
reading files from pretrained/dalle_bart_mega
initializing MinDalleTorch
loading encoder
loading decoder
Traceback (most recent call last):
  File "/home/user/min-dalle/image_from_text.py", line 68, in <module>
    generate_image(
  File "/home/user/min-dalle/image_from_text.py", line 48, in generate_image
    image_generator = MinDalleTorch(is_mega, sample_token_count)
  File "/home/user/min-dalle/min_dalle/min_dalle_torch.py", line 60, in __init__
    self.decoder = self.decoder.cuda()
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 706, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 595, in _apply
    module._apply(fn)
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 595, in _apply
    module._apply(fn)
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 595, in _apply
    module._apply(fn)
  [Previous line repeated 1 more time]
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 618, in _apply
    param_applied = fn(param)
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 706, in <lambda>
    return self._apply(lambda t: t.cuda(device))
RuntimeError: HIP out of memory. Tried to allocate 32.00 MiB (GPU 0; 7.98 GiB total capacity; 7.92 GiB already allocated; 58.00 MiB free; 7.93 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_HIP_ALLOC_CONF
$ git checkout 1ef9b0b
Previous HEAD position was ed91ab4 refactored to load models once and run multiple times
HEAD is now at 1ef9b0b added mega to colab
$ python image_from_text.py --text="alien life" --mega --torch --seed 100
Namespace(mega=True, torch=True, text='alien life', seed=100, image_path='generated', image_token_count=256)
parsing metadata from ./pretrained/dalle_bart_mega
tokenizing text
['Ġalien']
['Ġlife']
text tokens [0, 8925, 742, 2]
loading torch encoder
encoding text tokens
loading torch decoder
sampling image tokens
detokenizing image
MIOpen(HIP): Warning [SQLiteBase] Missing system database file: gfx900_64.kdb Performance may degrade. Please follow instructions to install: https://github.com/ROCmSoftwarePlatform/MIOpen#installing-miopen-kernels-package
saving image to generated.png

[EeveeTea]

Works flawlessly, and looks to be faster too. Thanks for looking into it!

[kuprel]

Yeah if you’re generating images in a loop you should initialize the model once with MinDalleTorch and then call its method generate_image with a seed and text each time you want another image. The command line script is inefficient to loop over because it loads the entire model every time. The colab shows how to initialize the model then generate