BAM and PAM training on LEVIR-CD Dataset Memory Consumption Issue

Hello, I’ve read all issues before ask your advice in resolution this issue with me.

Firsltly thanks for sharing this work with people, I’ve learned many thinks from this project and continue learning.

I’ve downloaded pretraining weight of PAM from link attached on readme file for this project. Then run demo.py with my sample satellite images (1.5m in pixel). Result is good, but not excellent. Possible you trained your model with 0.5m in pixel, so firstly I’ve started to test train methods on LEVIR-CD Dataset with future planning with my own dataset. Base Method Learning done without problem. But BAM and PAM ask big memory(256GB).

So when I run code: python3 ./train.py --save_epoch_freq 1 --angle 15 --dataroot ../DATASET/train --val_dataroot ../DATASET/val --name LEVIR-CDFA_BAM2 --lr 0.001 --model CDFA --SA_mode BAM --batch_size 8 --load_size 256 --crop_size 256 --preprocess rotate_and_crop

As you see, I’ve 16GB Memory (Hosting provider AWS - with NVIDIA Tesla T4 VGA. So in another issues I’ve learn that, I can pass --ds 4 parameter - self attention module down-sample rate. It consumes 5GB memory, but as was expected demo results is not accurate. Your Pretrained model is better than my with --ds 4 parameter.

So my question is, how can I optimize training without downsample rate and work around memory consumption issue. As I’ve read from your paper

We tested our methods on a desktop PC equipped with an Intel i7-7700K CPU and an NVIDIA GTX 1080Ti graphic card. We used GPU to accelerate the training and testing process.

Possible you have some advises, to reach same model by quality as You shared via link? Possible PC params or Method parameters or input images parameters.

Thank you in advance

[tan90du-sx]

加一，我降低ds以后，效果很差。。还不如base

[johncse1]

Dear @loviji
I am getting the same IndexError. I too cropped the dataset by making each image of size 256x256 How can I make label maps bit depth to be equal to 8. Help me out.