Plotter memory leak due to not clearing up

@banesullivan I saw this was closed in 2019 however I don’t believe the whole problem has been addressed.

Just Running and destroying a pv.Plotter Instance in pyvista 0.32.1 (maybe later but untested) still produces a memory hogging for long running process.

Running the following code: (need to install memory_profiler)

from memory_profiler import memory_usage

import matplotlib.pyplot as plt
import pyvista as pv
import numpy as np
import gc

from time import sleep

gc.enable()


def plot_mem(mem, label=None, show=False):
    if label is None:
        label = 'Usage'
    mem = np.array(mem) - mem[0]
    if show:
        plt.close('all')
        plt.plot(np.arange(len(mem)) * 0.1, mem, label=label)
        plt.ylabel('MB')
        plt.xlabel('time (in s)')  # `memory_usage` logs every 100ms
        plt.legend()
        plt.title('memory usage')
        plt.show()
    print(f'Max: {mem.max():.2f} MB\nLast: {mem[-1]:.2f} MB')


def plot_pyvista():
    # sphere = pv.Sphere()
    for i in range(1000):
        plotter = pv.Plotter()
        plotter.close()

    gc.collect()
    sleep(0.01)


if __name__ == '__main__':
    mem_full = memory_usage((plot_pyvista, [], {}))
    plot_mem(mem_full, label='Full', show=True)

We can see that 70 Mib is held over time:

Running the following code to try and understand where the memory was being held:

import pyvista as pv

import pandas as pd
from pprint import pprint

from collections import defaultdict
from gc import get_objects
import gc

import sys
before = defaultdict(int)
before_size = defaultdict(int)
after = defaultdict(int)
after_size = defaultdict(int)
for i in get_objects():
    before[type(i)] += 1
    before_size[type(i)] += sys.getsizeof(i)

for i in range(1000):
    plotter = pv.Plotter(off_screen=False)
    plotter.close()

not_collected = gc.collect()


for i in get_objects():
    after[type(i)] += 1
    after_size[type(i)] += sys.getsizeof(i)

df = pd.DataFrame([(str(k), after[k] - before[k], (after_size[k] - before_size[k]))
                  for k in after if after[k] - before[k]], columns=['class', 'count', 'size'])

df = df.groupby('class').sum().reset_index()
pprint(df.sort_values('size').tail(25))
print('\n\n')
pprint(df[['pyvista' in df['class'].astype(str)[i] for i in range(len(df))]])
print('\n\n')
# print(f'{df.size.sum() / 1024:.2f} MB')
print(f"total objects: {df['count'].sum()}")
print(f'not collected objects: {not_collected}')

Results:

                                                class  count     size
1   <class '_frozen_importlib_external.ExtensionFi...      1       48
11                                   <class 'module'>      1       72
9                         <class 'member_descriptor'>      2      128
2                <class 'builtin_function_or_method'>     17     1224
7                         <class 'getset_descriptor'>     35     2240
25                       <class 'wrapper_descriptor'>     94     6768
22  <class 'vtkmodules.vtkCommonCore.method_descri...    385    27720
3                                      <class 'cell'>    842    33680
16     <class 'pyvista.plotting.renderers.Renderers'>   1000    48000
13        <class 'pyvista.plotting.plotting.Plotter'>   1000    48000
17  <class 'pyvista.plotting.scalar_bars.ScalarBars'>   1000    48000
14  <class 'pyvista.plotting.render_window_interac...   1000    88000
10                                   <class 'method'>   2000   128000
23                                <class 'weakproxy'>   2000   144000
24                                  <class 'weakref'>   2016   145152
6                         <class 'functools.partial'>   2000   160000
12           <class 'pyvista.plotting.camera.Camera'>   2000   176000
15       <class 'pyvista.plotting.renderer.Renderer'>   2000   176000
20                                    <class 'tuple'>   5446   284624
18              <class 'pyvista.themes.DefaultTheme'>   1000   320000
5                                  <class 'function'>   2854   388144
8                                      <class 'list'>  10994   799344
21                                     <class 'type'>    999  1063024
19                                      <class 'set'>   2000  1456000
4                                      <class 'dict'>   8990  3844632



                                                class  count    size
12           <class 'pyvista.plotting.camera.Camera'>   2000  176000
13        <class 'pyvista.plotting.plotting.Plotter'>   1000   48000
14  <class 'pyvista.plotting.render_window_interac...   1000   88000
15       <class 'pyvista.plotting.renderer.Renderer'>   2000  176000
16     <class 'pyvista.plotting.renderers.Renderers'>   1000   48000
17  <class 'pyvista.plotting.scalar_bars.ScalarBars'>   1000   48000
18              <class 'pyvista.themes.DefaultTheme'>   1000  320000



total objects: 49677
not collected objects: 7000

It appears a bunch of pyvista objects are not garabge collected I am guessing this is because there are still references between them.

by changing the loop to:

for i in range(1000):
    plotter = pv.Plotter(off_screen=False)
    plotter.close()

    pv.plotting._ALL_PLOTTERS.pop(plotter._id_name)
    plotter.renderers = None
    delattr(plotter, 'renderers')
    plotter._theme = None
    del plotter

Results:

                                                class  count     size
8                                      <class 'list'>     -6    -8656
0              <class '_frozen_importlib.ModuleSpec'>      1       48
1   <class '_frozen_importlib_external.ExtensionFi...      1       48
16                                <class 'weakproxy'>      1       72
11                                   <class 'module'>      1       72
9                         <class 'member_descriptor'>      2      128
2                <class 'builtin_function_or_method'>     17     1224
7                         <class 'getset_descriptor'>     35     2240
18                       <class 'wrapper_descriptor'>     94     6768
15  <class 'vtkmodules.vtkCommonCore.method_descri...    385    27720
3                                      <class 'cell'>    842    33680
12  <class 'pyvista.plotting.render_window_interac...   1000    88000
10                                   <class 'method'>   2000   128000
17                                  <class 'weakref'>   2016   145152
6                         <class 'functools.partial'>   2000   160000
13                                    <class 'tuple'>   5446   284624
5                                  <class 'function'>   2854   388144
4                                      <class 'dict'>   1990   639904
14                                     <class 'type'>    999  1062728



                                                class  count   size
12  <class 'pyvista.plotting.render_window_interac...   1000  88000



total objects: 19678
not collected objects: 7120

The only lingering object appears to be <class 'pyvista.plotting.render_window_interactor._style_factory.<locals>.CustomStyle'> I managed to trace this down to the init of CustomStyle and in particular the self.AddObserver calls, commenting these lines (and the whole class …) seems to still provide a working pyvista …

Running the inital memory plot with all these modifications (commented out CustomStyle) I get:

4 Mb so still stuff hanging around but more reasonable.

I am guessing most of the changes can be added to Plotter.close just not sure how to handle the CustomStyle

_Originally posted by @WesleyTheGeolien in https://github.com/pyvista/pyvista/issues/482#issuecomment-1104821364_

[akaszynski]

Thanks for your detailed analysis, we’ll look into this.

[courtney7]

I am running a web service to dynamically render an image using pyvista and serve it to a user. I’ve found it to be a really powerful module to work with. Once I have got my image data using _, image_data = pl.show(screenshot=True, interactive=False, return_img=True, return_cpos=True, full_screen=False), I then call pyvista.close_all(). That fixed a memory leak in the long-running process I was experiencing with pyvista 0.34.1. I can then work with image_data to return the image to the user.

This assumes that it is safe to clear all the plots.