Potential memory leak in MultipartWriter._upload_next_part

MultipartWriter._upload_next_part Memory leak

the MultipartWriter._upload_next_part function upload the bytesIo from Memory to s3(by upload[CreateMultipartUpload]),but do not clean bytesIO (in Memory). i use the smart_open.open() (gz file) upload to s3,the Memory Keep increasing。 I check the code and focus the MultipartWriter._upload_next_part() :

        self._buf.seek(0)
        part = self._mp.Part(part_num)
        upload = _retry_if_failed(functools.partial(part.upload, Body=self._buf))
        self._parts.append({'ETag': upload['ETag'], 'PartNumber': part_num})
        logger.debug("upload of part #%i finished" % part_num)

        self._total_parts += 1
        self._buf = io.BytesIO()

after upload the bytesIO, not clean the Memory.

bug fix code

        self._buf.seek(0)
        part = self._mp.Part(part_num)
        upload = _retry_if_failed(functools.partial(part.upload, Body=self._buf))
        self._parts.append({'ETag': upload['ETag'], 'PartNumber': part_num})
        logger.debug("upload of part #%i finished" % part_num)

       
        self._buf.truncate()# clean the Memory , avoid Memory leak
        self._total_parts += 1
        self._buf = io.BytesIO()

Versions

python 3.7 smart_open 2.0.0

Checklist

https://docs.python.org/zh-cn/3/library/io.html

[SchopenhauerZhang]

wouldn’t the garbage collector take care of this? em， i use smart_open in High concurrency and high throughput distributed system. wait the garbage collector(python’s) to deal it is not good idea. I suppose to recycle the memory by self. I will try to recycle the memory by self, later i write the result at here. Thanks!

[mpenkov]

@cnmoise Can you reproduce the problem with the newest version of smart_open? We’ve reworked the buffering in the S3 submodule significantly, and it’s likely the problem is no longer there.