Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

s3 bucket to bucket copy operation failed, 1 GB files - Linking did not work

See original GitHub issue

Environment info

NooBaa Version: master-20210607
Platform: OCP 4.6.16

Actual behavior

Copy operation from one bucket to another failed

Expected behavior

Copy operation should not fail

Steps to reproduce

Created 2 buckets - bucket-1, bucket-21.
Uploaded multiple 1GB files to s3://bucket-1
Attempted copy operation from s3://bucket-1 to s3://bucket-21

Few files got copied but got failure for others:

[root@hpo-node1 akshat]# s3u1 ls s3://bucket-1/
urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3-noobaa.apps.ocp-akshat-1.cp.fyre.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2021-06-08 21:15:06 1024000000 small_chunk_1
2021-06-08 21:15:28 1024000000 small_chunk_1._read
2021-06-08 21:15:12 1024000000 small_chunk_10
2021-06-08 21:15:26 1024000000 small_chunk_10._read
2021-06-08 21:15:27 1024000000 small_chunk_1_read
2021-06-08 21:16:15 1024000000 small_chunk_2
2021-06-08 21:16:18 1024000000 small_chunk_2._read
2021-06-08 21:16:53 1024000000 small_chunk_3
2021-06-08 21:16:25 1024000000 small_chunk_3._read
2021-06-08 21:16:08 1024000000 small_chunk_4
2021-06-08 21:17:12 1024000000 small_chunk_4._read
2021-06-08 21:17:25 1024000000 small_chunk_5
2021-06-08 21:17:36 1024000000 small_chunk_5._read
2021-06-08 21:17:44 1024000000 small_chunk_6
2021-06-08 21:18:36 1024000000 small_chunk_6._read
2021-06-08 21:19:13 1024000000 small_chunk_7
2021-06-08 21:19:05 1024000000 small_chunk_7._read
2021-06-08 21:19:22 1024000000 small_chunk_8
2021-06-08 21:19:29 1024000000 small_chunk_8._read
2021-06-08 21:19:39 1024000000 small_chunk_9
2021-06-08 21:19:37 1024000000 small_chunk_9._read
[root@hpo-node1 akshat]# 
[root@hpo-node1 akshat]# 
[root@hpo-node1 akshat]# s3u1 ls s3://bucket-21/
urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3-noobaa.apps.ocp-akshat-1.cp.fyre.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2021-06-08 21:30:55 1024000000 small_chunk_8
2021-06-08 21:30:55 1024000000 small_chunk_8._read
2021-06-08 21:30:55 1024000000 small_chunk_9
2021-06-08 21:30:55 1024000000 small_chunk_9._read

Command used to copy files: s3u1 cp --recursive s3://bucket-1 s3://bucket-21/

Errors:

copy failed: s3://bucket-1/small_chunk_9 to s3://bucket-21/small_chunk_9 An error occurred (NoSuchKey) when calling the CompleteMultipartUpload operation (reached max retries: 2): The specified key does not exist.
copy failed: s3://bucket-1/small_chunk_8 to s3://bucket-21/small_chunk_8 An error occurred (NoSuchKey) when calling the CompleteMultipartUpload operation (reached max retries: 2): The specified key does not exist.
copy failed: s3://bucket-1/small_chunk_9._read to s3://bucket-21/small_chunk_9._read An error occurred (NoSuchKey) when calling the CompleteMultipartUpload operation (reached max retries: 2): The specified key does not exist.

More information - Screenshots / Logs / Other output

Jun-9 4:29:35.711 [Endpoint/9]   [LOG] CONSOLE:: update_endpoint_stats. namespace_stats = []
Jun-9 4:30:05.205 [Endpoint/9]   [LOG] CONSOLE:: NamespaceFS: abort_object_upload /nsfs/nsfs-nsr-1/bucket-21/.noobaa-nsfs_60c0406ba0157e00233de2dc/multipart-uploads/f989c13a-82c4-40e7-b1ba-71162a3
def85
Jun-9 4:30:05.897 [Endpoint/9] [ERROR] CONSOLE::  [Error: No such file or directory] { code: 'ENOENT' }
Jun-9 4:30:05.897 [Endpoint/9] [ERROR] CONSOLE::  [Error: No such file or directory] { code: 'ENOENT' }
Jun-9 4:30:05.995 [Endpoint/9] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><R
esource>/bucket-21/small_chunk_4._read?uploadId=f989c13a-82c4-40e7-b1ba-71162a3def85</Resource><RequestId>kpoz3eu6-6rwwuk-1aal</RequestId></Error> POST /bucket-21/small_chunk_4._read?uploadId=f989
c13a-82c4-40e7-b1ba-71162a3def85 {"accept-encoding":"identity","user-agent":"aws-cli/2.1.29 Python/3.8.8 Linux/4.18.0-305.3.1.el8_4.x86_64 exe/x86_64.rhel.8 prompt/off command/s3.cp","x-amz-date":
"20210609T042933Z","x-amz-content-sha256":"cc82905fa5a8bc75908709678e6972231fe14ce147ac9ef3611ec94ac4575f9f","authorization":"AWS4-HMAC-SHA256 Credential=YrpJ6lYpmcU6u8ODljQG/20210609/us-east-1/s3
/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=fad4887a005b0cd9e9fe190aadbbc283719e3d4af452b41385d8c04b97cf428e","content-length":"11641","host":"s3-noobaa.apps.ocp-a
kshat-1.cp.fyre.ibm.com","x-forwarded-host":"s3-noobaa.apps.ocp-akshat-1.cp.fyre.ibm.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.17.66.193;host=s3-noobaa.apps.ocp
-akshat-1.cp.fyre.ibm.com;proto=https","x-forwarded-for":"10.17.66.193"} Error: No such file or directory
Jun-9 4:30:05.996 [Endpoint/9] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><R
esource>/bucket-21/small_chunk_4._read?uploadId=f989c13a-82c4-40e7-b1ba-71162a3def85</Resource><RequestId>kpoz2p77-313rnn-1076</RequestId></Error> POST /bucket-21/small_chunk_4._read?uploadId=f989
c13a-82c4-40e7-b1ba-71162a3def85 {"accept-encoding":"identity","user-agent":"aws-cli/2.1.29 Python/3.8.8 Linux/4.18.0-305.3.1.el8_4.x86_64 exe/x86_64.rhel.8 prompt/off command/s3.cp","x-amz-date":
"20210609T042901Z","x-amz-content-sha256":"cc82905fa5a8bc75908709678e6972231fe14ce147ac9ef3611ec94ac4575f9f","authorization":"AWS4-HMAC-SHA256 Credential=YrpJ6lYpmcU6u8ODljQG/20210609/us-east-1/s3
/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=886bd10e1ca6928f0e01cb53ed17f8675de6e6d42d8ab351484e2dbf3f25c192","content-length":"11641","host":"s3-noobaa.apps.ocp-a
kshat-1.cp.fyre.ibm.com","x-forwarded-host":"s3-noobaa.apps.ocp-akshat-1.cp.fyre.ibm.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.17.66.193;host=s3-noobaa.apps.ocp
-akshat-1.cp.fyre.ibm.com;proto=https","x-forwarded-for":"10.17.66.193"} Error: No such file or directory
Jun-9 4:30:12.901 [Endpoint/9]   [LOG] CONSOLE:: NamespaceFS: abort_object_upload /nsfs/nsfs-nsr-1/bucket-21/.noobaa-nsfs_60c0406ba0157e00233de2dc/multipart-uploads/c33a6269-c625-4d62-9d9c-1b34258
23bd3
Jun-9 4:30:13.298 [Endpoint/9] [ERROR] CONSOLE::  [Error: No such file or directory] { code: 'ENOENT' }

Issue Analytics

State:
Created 2 years ago
Comments:24 (15 by maintainers)

Top GitHub Comments

1reaction

liranmaudacommented, Jun 10, 2021

Regarding the Copy: In the description of the issue we see that the copy is from one bucket to another

Actual behavior

Copy operation from one bucket to another failed

We can see in the block below that we are checking if we consider this as a server side copy (according to the below parameter):

https://github.com/noobaa/noobaa-core/blob/052040a8b826e567d0d60847f741ff2e4a9554d6/src/sdk/namespace_fs.js#L158-L163

If we are not considering this as a server side copy we will do plain upload.

0reactions

nimrod-beckercommented, Jul 5, 2021

Closing the issue as we don’t seem to have something to fix in our code, we can open a new one with a more accurate reason if we need it

Top Results From Across the Web

Amazon S3 replication failure reasons

SrcBucketInvalidRegion, Source bucket is not in same AWS Region as specified by the Batch Operations job. This error is specific to Batch Replication....

AWS cli throws error when copying large files - Stack Overflow

It only copys one file at once. You can grab a list of objects in the bucket using s3 command ls or s3api...

syncing many large files between s3 buckets occasionally ...

syncing many large files between s3 buckets occasionally results in a copy failed due to a CompleteMultipartUpload operation: Access Denied ...

Resolve issues with uploading large files in Amazon S3

I'm trying to upload a large file (1 GB or larger) to Amazon Simple Storage Service (Amazon S3) using the console. However, the...

8 Must-Know Tricks to Use S3 More Effectively in Python

If you did not configure your S3 bucket to allow public access, you will receive ... Uploading large files to S3 at once...