Possible memory leaks in Cython 3.0a1—a4

I’m testing gevent with Cython 3. gevent has a combination of Cython code in .pyx/pxd and pure-Python code annotated and compiled with .pxd files. (Which is awesome, BTW. Thanks again for a great project!)

gevent’s test suite has a job that runs all the tests and watches for memory leaks. It basically does this by using gc.get_objects() to count all the objects extent, run a particular test, and count the objects again. If the counts don’t match, this process is repeated a few times to see if a pattern emerges. When it’s been run enough times to see that a particular type of object is always gaining new instances after a test run, the test is declared to leak and is failed.

Using Cython 0.29.17, this test job passes.

The test job with Cython 3.0a4 fails; many tests report leaks. For example:

  FAIL: test_raises_timeout_Timeout_exc_customized (__main__.TestGet)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 214, in wrapper
      return _RefCountChecker(self, method)(args, kwargs)
    File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 192, in __call__
      while self._check_deltas(growth):
    File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 168, in _check_deltas
      raise AssertionError('refcount increased by %r\n%s' % (deltas, diff))
  AssertionError: refcount increased by [9, 1168, 2, 2, 2]
  RuntimeError        6        +1
  Timeout            10        +1

That’s showing that the last three runs of this test resulted in two new objects each time; one new RuntimeError and one new Timeout (a kind of exception). I thought at first that maybe it was just exceptions leaking since the first reports are just of exceptions, but later on we see a variety of objects leaking:

840  FAIL: test_unordered (__main__.TestErrorInIterator)
841  ----------------------------------------------------------------------
842  Traceback (most recent call last):
843    File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 214, in wrapper
844      return _RefCountChecker(self, method)(args, kwargs)
845    File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 192, in __call__
846      while self._check_deltas(growth):
847    File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 168, in _check_deltas
848      raise AssertionError('refcount increased by %r\n%s' % (deltas, diff))
849  AssertionError: refcount increased by [17, 1187, 10, 10, 10]
850  list               2061        +2
851  deque                 5        +1
852  ThreadPool            5        +1
853  instancemethod      116        +1
854  set                 291        +1
855  Queue                 5        +1
856  Semaphore             5        +1
857  fork                  5        +1
858  _Condition            7        +1

I’ve been using language_level=3str in most places for some time already. The only other thing that caught my eye as a possible cause was the change to make binding True by default. But I changed that directive in my setup.py and tested locally and still get the failures.

Locally I’ve tested 3.0a1 through 3.0a4, inclusive, and get the failures on all versions. (I also tried git master as-of now—I think—and have the same issues.)

Next, I tried not using Cython compiled versions of the pure-Python modules, keeping just the compiled .pyx files, and locally the tests pass again (let me know if that wasn’t clear). So it appears to have something to do with that mode?

I haven’t tried to create a smaller example yet because I was hoping the problem might just be obvious 😃 Please let me know what other info I can provide or how I can help track this down.

[jamadden]

If I make that change in ModuleSetupCode.c, all gevent’s leak tests pass.

[scoder]

Thanks for the report and the investigations @jamadden.