Unable to push images > 545MB with https

Hi, I’m having trouble pushing larger images through the GUI, sregistry-cli, and singularity when https is enabled. I see there have been similar issues resolved in the past, I’ve gone through those threads multiple times to try out different suggestions but haven’t had any success. I’m not sure if this is a bug or a misconfiguration on my part. When I change SREGISTRY_REGISTRY_BASE to use http everything works.

singularity version: 3.5.3 sregistry version: 1.1.22 sregistry-cli version:0.2.35

Success with an image < 545MB

$ sregistry pull n8/hello:latest
[client|registry] [database|sqlite:////home/n/.singularity/sregistry.db]
Progress |===================================| 100.0% 
[container][new] n8/hello:latest@36e003395e3cef288da37ee1826ec2b89e15b3ac7fa9b70be8a807f69562881b
Success! /home/n/.singularity/shub/n8/hello/latest@36e003395e3cef288da37ee1826ec2b89e15b3ac7fa9b70be8a807f69562881b.sif

$ sregistry push --name n8/hello:200319 .singularity/shub/n8/hello/latest@36e003395e3cef288da37ee1826ec2b89e15b3ac7fa9b70be8a807f69562881b.sif 
[client|registry] [database|sqlite:////home/n/.singularity/sregistry.db]
[1. Collection return status 200 OK]
[================================] 59/59 MB - 00:00:00
[Return status 200 Upload Complete]

Failure with a large image

The upload dies when it reaches around 554MB

@ganymede:~$ sregistry pull n8/fmri4:latest
[client|registry] [database|sqlite:////home/n/.singularity/sregistry.db]
Progress |===================================| 100.0% 
[container][new] n8/fmri4:latest@92278b7c046c0acf0952b3e1663b8abb819c260e8a96705bad90833d87ca0874
Success! /home/n/.singularity/shub/n8/fmri4/latest@92278b7c046c0acf0952b3e1663b8abb819c260e8a96705bad90833d87ca0874.sif

$ sregistry push --name n8/fmri:200319 /home/n/.singularity/shub/n8/fmri4/latest@92278b7c046c0acf0952b3e1663b8abb819c260e8a96705bad90833d87ca0874.sif
[client|registry] [database|sqlite:////home/n/.singularity/sregistry.db]
[1. Collection return status 200 OK]
Traceback (most recent call last): 545/4639 MB - 00:21:52
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1230, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1276, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1225, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1043, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.8/http/client.py", line 965, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1230, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1276, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1225, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1043, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.8/http/client.py", line 965, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/n/.local/bin/sregistry", line 10, in <module>
    sys.exit(main())
  File "/home/n/.local/lib/python3.8/site-packages/sregistry/client/__init__.py", line 391, in main
    main(args=args, parser=parser, extra=extra)
  File "/home/n/.local/lib/python3.8/site-packages/sregistry/client/push.py", line 33, in main
    cli.push(path=image, name=args.name, tag=args.tag)
  File "/home/n/.local/lib/python3.8/site-packages/sregistry/main/registry/push.py", line 96, in push
    r = requests.post(url, data=monitor, headers=headers)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

nginx logs

- - [19/Mar/2020:20:17:27 +0000] "POST /api/upload/chunked_upload HTTP/1.1" 200 10 "-" "python-requests/2.22.0" "-"
- - [19/Mar/2020:20:20:22 +0000] "POST /upload HTTP/1.1" 400 0 "-" "python-requests/2.22.0" "-"

uwsgi logs

200319 fmri n8 SREGISTRY-HMAC-SHA256 Credential=push/<removed>,Signature=7346e058a5e48b271d2ff33996c244c0b090cdaa225a57513dc01f5ea2205932 {'collection': 'n8', 'tag': '200319', 'name': 'fmri'}
push|n8|20200319T20Z|fmri|200319|
[pid: 40|app: 0|req: 13/35] () {42 vars in 720 bytes} [Thu Mar 19 15:17:26 2020] POST /api/upload/chunked_upload => generated 10 bytes in 45 msecs (HTTP/1.1 200) 3 headers in 100 bytes (1 switches on core 3)

config.py

DOMAIN_NAME = 'https://mydomain.org'
DOMAIN_NAME_HTTP = 'https://mydomain.org'
DOMAIN_NAKED = DOMAIN_NAME_HTTP.replace("http://", "")
DATA_UPLOAD_MAX_MEMORY_SIZE = None

nginx.conf

server {
  listen                *:80;
  listen              443 ssl;
  server_name         mydomain.org;
  ssl_certificate     /etc/ssl/cert.pem;
  ssl_certificate_key /etc/ssl/cert.key;
  client_max_body_size 10024M;
  client_body_buffer_size 10024M;
  client_body_timeout 120;
  add_header X-Clacks-Overhead "GNU Terry Pratchett";
  add_header X-Clacks-Overhead "GNU Terry Pratchet";
  add_header Access-Control-Allow-Origin *;
  add_header 'Access-Control-Allow-Credentials' 'true';
  add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
  add_header 'Access-Control-Allow-Headers' 'Authorization,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
  location /images {
    alias /var/www/images;
  }
  location ~* \.(php|aspx|myadmin|asp)$ {
    deny all;
  }
  location / {
    include /etc/nginx/uwsgi_params.par;
    uwsgi_pass uwsgi:3031;
    uwsgi_max_temp_file_size 10024m;
    # troubleshooting large upload issue
    uwsgi_read_timeout 36000;
    client_max_body_size 5g;
  }
  location /static {
    alias /var/www/static;
  }
  location /upload {
        upload_pass   /api/uploads/complete/;
        upload_store /var/www/images/_upload 1;        
        upload_store_access user:rw group:rw all:rw;
        upload_set_form_field $upload_field_name.name "$upload_file_name";
        upload_set_form_field $upload_field_name.content_type "$upload_content_type";
        upload_set_form_field $upload_field_name.path "$upload_tmp_path";
        upload_aggregate_form_field "$upload_field_name.md5" "$upload_file_md5";
        upload_aggregate_form_field "$upload_field_name.size" "$upload_file_size";
        upload_pass_form_field "^submit$|^description$";
        upload_pass_form_field "^SREGISTRY_EVENT$";
        upload_pass_form_field "^collection$";
        upload_pass_form_field "^name$";
        upload_pass_form_field "^tag$";
        upload_cleanup 400-599;
        # troubleshooting large upload issue
        uwsgi_read_timeout 36000;
        client_max_body_size 5g;
    }
}

[n8wood-brown]

@vsoch - I just doubled the memory to 32GB and that didn’t help. I’ll look into the mulipart uploads link you posted.

Thank you so much for the attention you’ve put into this!

[vsoch]

I definitely think so - the way that I usually test and develop is on localhost, and I compile Singularity to allow for http:// for the remote endpoint (the link is provided in the docs on this page) https://singularityhub.github.io/sregistry/docs/client#singularity-push. Then you can use 127.0.0.1 as the remote, and see if the error reproduces there. If that works, then it’s some issue with your setup.