DAL object leaks memory

I have an application that runs on CherryPy (v8.1.2) on Ubuntu (14.04.5 LTS) and python 2.7.6. I use standalone pyDAL (v16.09) to connect to my MySQL DB on AWS RDS. I’m seeing a constant memory leak on my servers which causes the process to use up 100% of RAM in a few hours and eventual die a painful death (no SWAP enabled; enabling SWAP pushes back certain death a little).

I have managed to reproduce the leak with a very small loop that repeatedly opens and closes the DAL object. I confirmed that memory continues to grow using htop and will eventually end up using 100% of RAM. I thought it was an MySQL adapter specific problem but I was able to repro it with the default sqlite adapter as well.

from pydal import DAL

def open_close_dal():
   sql = DAL()
   sql.close()

if __name__ == '__main__':
   while True:
      open_close_dal()

I read some really old threads on this issue that suggested calling sql._adapter.close() or sql._adapter.close_all_instances() or del sql but it doesn’t seem to make a difference. I’ve tried tracking down the memory leak and managed to confirm that it is the guts of the DAL that are leaking. One iteration of the loop, with pympler.tracker enabled reports these objects created:

                                       types |   # objects |   total size
============================================ | =========== | ============
                                        dict |         672 |    423.38 KB
                                        list |        3525 |    354.35 KB
                                         str |        2851 |    162.49 KB
             <class 'collections.OrderedDict |         129 |    135.02 KB
                                        code |         585 |     73.12 KB
          <class 'pydal.dialects.MetaDialect |          32 |     28.25 KB
            <class 'pydal.parsers.MetaParser |          17 |     15.01 KB
                                        type |          16 |     14.12 KB
  <class 'pydal.representers.MetaRepresenter |          16 |     14.12 KB
                                       tuple |         168 |     12.85 KB
          <class 'pydal.dialects.sqltype_for |         178 |     11.12 KB
                                     weakref |          89 |      7.65 KB
                                         int |         258 |      6.05 KB
         <class 'pydal.representers.for_type |          49 |      3.06 KB
                             function (wrap) |          25 |      2.93 KB

[gi0baro]

@toorsukhmeet ok found the leak.

@mdipierro this is because every DAL instance create data into THREAD_LOCAL variable and never remove the contents.

I’ve tested this:

def run_leak(n=100):
    for _ in xrange(0, n):
        db = DAL('sqlite:memory')
        db.close()

with some profiling on the memory usage:

(Pdb) run_leak()
(Pdb) objgraph.show_growth()
dict            1722       +90
list            1775       +90
OrderedDict      312       +90
(Pdb) run_leak()
(Pdb) objgraph.show_growth()
dict            1807       +85
list            1860       +85
OrderedDict      397       +85
(Pdb) run_leak()
(Pdb) objgraph.show_growth()
dict            1892       +85
list            1945       +85
OrderedDict      482       +85

and is quite obvious the THREAD_LOCAL is increasing due to connections:

>>> from pydal.connection import THREAD_LOCAL
>>> dir(THREAD_LOCAL)
['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_pydal_connection_4390457232_66113', '_pydal_cursors_4390457232_66113', '_pydal_db_instances_', '_pydal_db_instances_zombie_']
>>> run_leak(100000)
>>> len(dir(THREAD_LOCAL))
68690
>>> dir(THREAD_LOCAL)[:30]
['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_pydal_connection_4381352592_66113', '_pydal_connection_4381352720_66113', '_pydal_connection_4381352848_66113', '_pydal_connection_4381352912_66113', '_pydal_connection_4388182992_66113', '_pydal_connection_4388822160_66113', '_pydal_connection_4388822288_66113', '_pydal_connection_4388872336_66113', '_pydal_connection_4390457168_66113', '_pydal_connection_4390457232_66113', '_pydal_connection_4390495376_66113', '_pydal_connection_4391236048_66113', '_pydal_connection_4391239056_66113', '_pydal_connection_4391239120_66113', '_pydal_connection_4391239184_66113']

So this is happening on web2py because it creates a DAL instance per-request. On weppy the DAL instance is the same across the processes and is initialized when the server start, this is why I missed the leak.

The proposal is to edit the DAL.close method to add some cleaning of the connection data on the THREAD_LOCAL object:

def close(self):
        self._adapter.close()
        if self._db_uid in THREAD_LOCAL._pydal_db_instances_:
            db_group = THREAD_LOCAL._pydal_db_instances_[self._db_uid]
            db_group.remove(self)
            if not db_group:
                del THREAD_LOCAL._pydal_db_instances_[self._db_uid]
        # HERE: some magic to clean connections and cursors too

@mdipierro what do you think?

[gi0baro]

@toorsukhmeet Will inspect this during the weekend