Broken Connection on successive calls using add_bytes

I am attempting to debug an issue wherein I am pushing the binary image data represented in Python byte string into IPFS using ipfshttpclient.

I have been able to distill this down to an example with two successive calls of add_bytes with large byte strings:

import ipfshttpclient
import os

bytes = bytearray(os.urandom(100000))

IPFS_API = ipfshttpclient.Client(f"/dns/localhost/tcp/5001/http")
res1 = IPFS_API.add_bytes(bytes)
res2 = IPFS_API.add_bytes(bytes)

…produces:

Traceback (most recent call last):
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/requests/adapters.py", line 472, in send
    low_conn.send(i)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 977, in send
    self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/http.py", line 251, in _do_request
    return requests.request(*args, **kwargs)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/requests_wrapper.py", line 244, in request
    return session.request(method=method, url=url, **kwargs)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/requests_wrapper.py", line 223, in request
    return super(Session, self).request(method, url, *args, **kwargs)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: [Errno 32] Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "pngpushtest.py", line 8, in <module>
    res2 = IPFS_API.add_bytes(bytes)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/utils.py", line 142, in wrapper
    res = cmd(*args, **kwargs)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/client/base.py", line 15, in wrapper
    result = func(*args, **kwargs)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/client/__init__.py", line 223, in add_bytes
    data=body, headers=headers, **kwargs)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/http.py", line 50, in wrapper
    return func(self, *args, **merged)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/http.py", line 370, in request
    files, headers, data, timeout=timeout)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/http.py", line 288, in _request
    timeout=timeout)
  File "/private/tmp/ipwb/ipwbve/lib/python3.7/site-packages/ipfshttpclient/http.py", line 255, in _do_request
    six.raise_from(exceptions.ConnectionError(error), error)
  File "<string>", line 3, in raise_from
ipfshttpclient.exceptions.ConnectionError: ConnectionError: [Errno 32] Broken pipe

Changing the above line to bytes = bytearray(os.urandom(10)), using 10 instead of 100000 does not produce this exception.

• Python 3.7.2
• ipfshttpclient 0.4.12

[ibnesayeed]

Thanks @machawk1, if the new version of this library works well with the new version of the go-ipfs then we can move on.

[ntninja]

@machawk1: Thanks for your tremendously valuable research here! While, as @ibnesayeed found out, adding the header should not be a problem anymore, it’s still an unnecessary performance penalty and I’m glad to here that this can be removed with go-IPFS v0.4.19+.

I will create a patch that removes the Connection: close flag for v0.4.19+ and would welcome your feedback on this.

The problem with 0.4.20 was that several API methods had returned broken or incomplete responses making it basically useless if you relied on any of these. Hence I decided to blacklist it altogether, rather than dealing with support queries that I can do nothing about anyways. (As such I don’t think it was related to this, but maybe played a part as well.)