Memory issue when training 1024 resolution

I’m trying to train a 1024x1024 database on a V100 GPU. I tried both the tensorflow version and the pytorch version. Despite setting batch-gpu to 1, the tensorflow version always run out of system RAM(after the first tick, system ram total 51gb), and the pytorch version alway run out of cuda memory(before the first tick).

Here are my training settings:

python run_network.py --train --metrics 'none' --gpus 0 --batch-gpu 1 --resolution 1024 \
 --ganformer-default --expname art1 --dataset 1024art

Also, I always encounter the warning: tcmalloc: large alloc

[BlueberryGin]

Btw, I’m currently running fine with only saving output images and not saving attention maps

[dorarad]

I’ll update the default options in accordance with that so the people won’t get memory issues. Thank you for the openning this issue!