Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Twice memory usage in `encode_multipart_formdata`

See original GitHub issue

Basically, if you wants to upload file like 1GB it will cost you 2GB RAM.

Filename: /Users/kyrylo/.virtualenvs/drsdk/lib/python2.7/site-packages/requests/packages/urllib3/filepost.py

Line #    Mem usage    Increment   Line Contents
================================================
    59    981.3 MiB      0.0 MiB   @profile
    60                             def encode_multipart_formdata(fields, boundary=None):
    61                                 """
    62                                 Encode a dictionary of ``fields`` using the multipart/form-data MIME format.
    63                             
    64                                 :param fields:
    65                                     Dictionary of fields or list of (key, :class:`~urllib3.fields.RequestField`).
    66                             
    67                                 :param boundary:
    68                                     If not specified, then a random boundary will be generated using
    69                                     :func:`mimetools.choose_boundary`.
    70                                 """
    71    981.3 MiB      0.0 MiB       body = BytesIO()
    72    981.3 MiB      0.0 MiB       if boundary is None:
    73    981.3 MiB      0.0 MiB           boundary = choose_boundary()
    74                             
    75   1920.7 MiB    939.4 MiB       for field in iter_field_objects(fields):
    76    981.3 MiB   -939.4 MiB           body.write(b('--%s\r\n' % (boundary)))    77                             
    78    981.3 MiB      0.0 MiB           writer(body).write(field.render_headers())
    79    981.3 MiB      0.0 MiB           data = field.data
    80                             
    81    981.3 MiB      0.0 MiB           if isinstance(data, int):
    82                                         data = str(data)  # Backwards compatibility
    83                             
    84    981.3 MiB      0.0 MiB           if isinstance(data, six.text_type):
    85                                         writer(body).write(data)
    86                                     else:
    87   1920.7 MiB    939.4 MiB               body.write(data)
    88                             
    89   1920.7 MiB      0.0 MiB           body.write(b'\r\n')
    90                             
    91   1920.7 MiB      0.0 MiB       body.write(b('--%s--\r\n' % (boundary)))
    92                             
    93   1920.7 MiB      0.0 MiB       content_type = str('multipart/form-data; boundary=%s' % boundary)
    94                             
    95   2000.1 MiB     79.4 MiB       return body.getvalue(), content_type

P.S. using this profiler https://pypi.python.org/pypi/memory_profiler