ResourceWarning: unclosed file

Problem description

I’m seeing a ResourceWarning: unclosed file warning when using context managers to open files/streams with smart_open.

Note: if I don’t run under unittest, or if I don’t gzip the file, resources seem to be closed correctly. I’m guessing unittest and smart_open are somehow not coordinating correctly on closing the layers.

Steps/code to reproduce the problem

My test script:

import unittest
from smart_open import open as smart_open


class MyTestCase(unittest.TestCase):
    def test_load(self):
        with smart_open('input.csv.gz') as fh:
            print("opened file")


unittest.main()

Invocation:

% echo -e 'col1,col2\nval1,val2\nval3,val4' | gzip > input.csv.gz 

% PYTHONTRACEMALLOC=1 python test_load.py
opened file
/Users/kwilliams/miniconda3/lib/python3.7/unittest/case.py:615: ResourceWarning: unclosed file <_io.BufferedReader name='input.csv.gz'>
  testMethod()
Object allocated at (most recent call last):
  File "/Users/kwilliams/git/dispatcher/rush-springs-simulations/venv/lib/python3.7/site-packages/smart_open/smart_open_lib.py", lineno 548
    fobj = io.open(parsed_uri.uri_path, mode)
.
----------------------------------------------------------------------
Ran 1 test in 0.007s

OK

Versions

Darwin-18.0.0-x86_64-i386-64bit
Python 3.7.3 (default, Mar 27 2019, 16:54:48) 
[Clang 4.0.1 (tags/RELEASE_401/final)]
smart_open 1.9.0

[piskvorky]

@kenahoo thanks for the clear and detailed report. I agree context managers should be closing handles, so that looks like a bug.

@mpenkov is busy ATM – any chance you could take a stab at this yourself?

[mpenkov]

Cannot reproduce on linux Python 3.10.6 and smart_open 6.1.0.