Support automatic abortion s3 multipartUpload after a timeout.

If a MultipartUpload request fails and the MultipartUpload request is not aborted, directory xxx_s3_multipart_tmp will be preserved, and future MultipartUpload requests will always error.

Describe the solution you’d like

1. Create a callable instance when a MultipartUpload request received which used to abort this MultipartUpload. Callable instance contains:

   1. upload file name (xxx)
   2. uploadId
   3. last_update_time (xxx_s3_multipart_tmp modify time or xxx modify time)

2. Create a ScheduledExecutorService instance used to schedule auto abort MultipartUpload callable instance.

   1. last_update_time + timeout = delay

3. When execute auto abort func:

   1. check xxx_s3_multipart_tmp if exist
   2. check xxx_s3_multipart_tmp dir id if equals uploadId
   3. check if xxx_s3_multipart_tmp modify time changed
   4. check xxx if exist and if xxx modify time changed

May caused problem

• Upload object part or finish MultipartUpload request may be distributed to different proxies, how to coordinate different proxies? Leader to do this?

Urgency

Normal

[ZacBlanco]

yes, we do need to make code changes – in fact there is a TODO in the code for it left by the original author of the S3 handler

[jffree]

Hey @jffree , is this answer you question?

Regarding

Upload object part or finish MultipartUpload request may be distributed to different proxies, how to coordinate different proxies? Leader to do this?

I don’t see a clear way to get around this if someone is using multiple proxies. Alluxio just does not allow multiple clients to upload+complete the same file. I think we will just have to document a restriction that the same proxy must be used during a multipart upload to properly complete the request. If a load balancer in front of the proxies is desired it should deterministically redirect the request to the correct proxy

I agree to explain how we implement it in the document, and the consistency problem should be guaranteed by the client.