question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Memory leak in PyTurboJPEG?

See original GitHub issue

Hi,

There seems to be a case of unmanaged memory in PyTurboJPEG. Memory is allocated during the image decode process, but not deallocated later (at least not immediately), and the memory usage of the process keeps building up.

As an example, please consider the following code:

from turbojpeg import TurboJPEG, TJPF_GRAY, TJSAMP_GRAY
import psutil
import os
import time
jpeg = TurboJPEG("/usr/lib/x86_64-linux-gnu/libturbojpeg.so.0")
import numpy

i = 0
PROCESS = psutil.Process(os.getpid())

buf = None
with open('some_image.jpg', 'rb') as fin:
    buf = fin.read()
for i in range(100):
    print 'iteration', i
    img_array = jpeg.decode(buf)
    print img_array.shape
    print 'memory = ', PROCESS.memory_info().rss // 1024

The output of this program looks like this:

...
iteration 43
(2160, 4096, 3)
memory =  806464
iteration 44
(2160, 4096, 3)
memory =  832576
iteration 45
(2160, 4096, 3)
memory =  858472
iteration 46
(2160, 4096, 3)
memory =  884340
iteration 47
(2160, 4096, 3)
memory =  910188
iteration 48
(2160, 4096, 3)
memory =  936316
iteration 49
(2160, 4096, 3)
memory =  962216
iteration 50
...

The memory usage keeps building up till more than a GB. It sometimes goes back to a low value, but again the buildup happens.

The other popular turbojpeg python wrapper – jpeg4py – doesn’t have this problem. However, I prefer using PyTurboJPEG because it offers a scaling factor in the decode function which speeds up things considerably for my use case. But the memory build up is causing problems as I have a tight memory budget.

What could be the reason and are there any known fixes/workarounds?

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
trdastidarcommented, Sep 24, 2019

Yes, the problem has been addressed. Thanks for your support.

0reactions
lilohuangcommented, Sep 23, 2019

The memory growing should be fixed. Please let me know if you still have question. Thank you.

Read more comments on GitHub >

github_iconTop Results From Across the Web

TurboJPEG: Fix potential memory leaks (Id3f734ab) - Gerrit
TurboJPEG: Fix potential memory leaks Referring to https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=746, it seems that the values of local buffer ...
Read more >
Memory leak in jpeg compression. Bug or my mistake?
In the jpeg compression there appears to be a memory leak. So the usage of memory increases continuously. The rgb24 and jpeg buffers...
Read more >
Memory leak in Python DBus server application when using ...
I've run Mattias' DBus test server under valgrind and gdb, and it leaks a small amount of memory each time the code handles...
Read more >
Diagnosing and Fixing Memory Leaks in Python - Fugue
Logic errors, CPU spikes, and memory leaks are inevitable, but a good debugger, CPU profiler, and memory profiler can make finding these ...
Read more >
Downgrade av to 8.1.0 to fix memory leak (#69833) · 32a10648a5 ...
"requirements": ["PyTurboJPEG==1.6.6", "av==9.0.0"],. "requirements": ["PyTurboJPEG==1.6.6", "av==8.1.0"],. "dependencies": ["http"],.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found