Multipart Uploads to S3 fail

I’m trying to upload files to S3 storage, I used the sample code for both client (tus-js-client) and server. If I use FileStore everything works fine, but as soon as I use S3Store I’m not able to upload files over 12MB. Here is the debug output (I’ve added ‘write’ and ‘getOffset’ logs to understand the flow):

tus-node-server [TusServer] handle: POST / +0ms
tus-node-server:stores:s3store bucket "s3-upload-bucket" exists +8s
tus-node-server:stores:s3store [ID] initializing multipart upload +0ms
tus-node-server:stores:s3store [ID] multipart upload created (...) +573ms
tus-node-server:stores:s3store [ID] saving metadata +0ms
tus-node-server:stores:s3store [ID] metadata file saved +658ms
tus-node-server [TusServer] handle: PATCH /files/ID +2s
tus-node-server:stores:s3store [ID] getOffset +13ms
tus-node-server:stores:s3store [ID] retrieving metadata +0ms
tus-node-server:stores:s3store [ID] metadata from s3 +0ms
tus-node-server:stores:s3store [ID] write +1s
tus-node-server:stores:s3store [ID] retrieving metadata +1ms
tus-node-server:stores:s3store [ID] metadata from cache +0ms
tus-node-server:stores:s3store [ID] finished uploading part #1 +3s
tus-node-server:stores:s3store [ID] getOffset +0ms
tus-node-server:stores:s3store [ID] retrieving metadata +0ms
tus-node-server:stores:s3store [ID] metadata from cache +0ms

The request stays pending and the upload does not complete unless I restart the server, in that case the upload will proceed for other 12MB. If I restart enough times the upload is completed successfully.

Please note that to achieve that result I had to add:

app.use('/upload', function(req, res, next) {
  req.socket.server.keepAliveTimeout = 100000000000;
  req.socket.server.timeout = 100000000000;
  next();
},uploadHandler);

otherwise node closes the connection as it recognises it is inactive and the browser resend the PATCH with upload-offset: 0 and I get a 409 error ([PatchHandler] send: Incorrect offset - 0 sent but file is 8*1024*1024)

Any idea on how I could fix this?

[saschbro]

looks like you defined different chunk sizes at client (8 mb) and server (12 mb). afaik chunk sizes have to be exactly the same at client and server otherwise uploads bigger than chunk size will not work correctly. you can set chunk size in tus-js-client by option chunkSize (see https://github.com/tus/tus-js-client#tusdefaultoptions).

[Acconut]

In early version of tusd, such a part would have been discarded and the client should try to upload a larger chunk in the next PATCH request.

Now, tusd stores these incomplete parts on S3 to not discard data. When a PATCH request begins, we first fetch this incomplete part and prepend it to the request body.