Out of memory during validation step

I got OOM error during validation step. Here is the log

0/10 * Epoch (train): 100% 32/32 [00:41<00:00,  2.39s/it, _fps=11.486, loss=0.898] 
0/10 * Epoch (valid):   3% 1/32 [00:01<00:46,  1.50s/it, _fps=21.531, loss=1.656]
Traceback (most recent call last):
....
RuntimeError: CUDA error: out of memory

My model uses only 80% of GPU during training. However, in validation step, It is out of memory. That is so weired since I though validation consums less memory than training. I am not sure it is normal or not. But I guess, probably, GPU does not have time to release GPU before going to validation step.

I also tried to add some callback to freeze GPU:

class FreeGPU(Callback):

    def on_stage_start(self, state):
        torch.cuda.empty_cache()

    def on_loader_start(self, state):
        torch.cuda.empty_cache()

    def on_loader_end(self, state):
        torch.cuda.empty_cache()

    def on_stage_end(self, state):
        torch.cuda.empty_cache()

    def on_epoch_start(self, state):
        torch.cuda.empty_cache()

    def on_epoch_end(self, state):
        torch.cuda.empty_cache()

It does not help at all. Do you have any ideas?. P/S: It is not the first time I face to this problem. The only thing I can prevent it is reducing the batch size. But, It hurts the performance.

[ngxbac]

Which version do you use? 19.03?

I am using 19.03-rc

PS. please, check that you have this grad check

You are right. I hotfix the library and this error has gone. I think it is the time to update my Catalyst 😢

[Scitator]

Which version do you use? 19.03?

PS. please, check that you have this grad check