Torch memory leak

It looks like there is a memory leak with Torch.

I tried to run tests/Test with different configurations and my findings are:

1) backend=Reference

Appears to run fine. It is slow but runs with constant memory consumption.

2) backend=Torch, device=CPU

Runs faster but starts swapping after a while and gets killed by the OOM killer after exhausting all memory and swap space:

net params: 1199882
Torch
Duration   |Iters| Ep|  Minib| Loss
0.00:00:04 |   1 | 1 | 1/937 | 2.336721e+000 🡾 New min
0.00:00:08 |   2 | 1 | 2/937 | 2.202248e+000 🡾 New min
0.00:00:11 |   3 | 1 | 3/937 | 1.961257e+000 🡾 New min
0.00:00:22 |   4 | 1 | 4/937 | 1.806348e+000 🡾 New min
0.00:01:41 |   5 | 1 | 5/937 | 1.367965e+000 🡾 New min
<killed>

3) backend=Torch, device=GPU

Also exhausts memory - this time on GPU:

net params: 1199882
Torch
Duration   |Iters| Ep|  Minib| Loss
0.00:00:02 |   1 | 1 | 1/937 | 2.316844e+000 🡾 New min
Unhandled exception. System.Runtime.InteropServices.ExternalException (0x80004005): CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 5.93 GiB total capacity; 4.85 GiB already allocated; 6.06 MiB free; 5.04 GiB reserved in total by PyTorch) (malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:289)

[dsyme]

@gbaydin FWIW for tests\Test on WIndows at the moment I see

• A quick rise to ~17GB after about 5 iterations, then 25GB after 8
• About 10 iterations completed then I hit Ctrl-C

Just in case it’s helpful.

I’ll start a separate thread about how we should test memory usage

[dsyme]

Great, thanks for confirming!