Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

nsfs: Unable to perform any operations in s3 buckets

See original GitHub issue

Environment info

NooBaa Version: 5.7
Platform: OpenShift 4.6

Actual behavior

I am facing issue to do any operations on buckets, I have created a Bucket ‘test-10’ manually by using s3 mb:

[root@hpo-nish1 s3-tests]# s3 mb s3://test-10
make_bucket: test-10

While doing any operation on this bucket, It is giving an error:

[root@hpo-nish1 s3-tests]# s3 ls s3://test-10
An error occurred (InternalError) when calling the ListObjectsV2 operation (reached max retries: 2): We encountered an internal error. Please try again.

I am finding that inspite of settings defined by chown and chmod for PVC’s of Filesystem and db-noobaa-db-0, we are not able to create buckets inside /nsfs. The endpoint log states error:

Mar-15 8:53:59.742 [Endpoint/13] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/test-10?list-type=2&amp;prefix=&amp;delimiter=%2F&amp;encoding-type=url</Resource><RequestId>kmaco7cd-e1bw0c-igk</RequestId></Error> GET /test-10?list-type=2&prefix=&delimiter=%2F&encoding-type=url {"host":"ocp-akshat-inf:8443","accept-encoding":"identity","user-agent":"aws-cli/2.1.27 Python/3.7.3 Linux/4.18.0-240.15.1.el8_3.x86_64 exe/x86_64.rhel.8 prompt/off command/s3.ls","x-amz-date":"20210315T085359Z","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","authorization":"AWS4-HMAC-SHA256 Credential=6IUhji5FNL3rn6Eafrj0/20210315/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d9dabf61e7d658a5f64cc37c07615cea4450349b7da6edec93c1b0938cc4bb14"} Error: ENOENT: no such file or directory, mkdir '/nsfs/test-10'
Mar-15 8:54:00.346 [Endpoint/13] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/test-10?list-type=2&amp;prefix=&amp;delimiter=%2F&amp;encoding-type=url</Resource><RequestId>kmaco7sp-78hy20-18x4</RequestId></Error> GET /test-10?list-type=2&prefix=&delimiter=%2F&encoding-type=url {"host":"ocp-akshat-inf:8443","accept-encoding":"identity","user-agent":"aws-cli/2.1.27 Python/3.7.3 Linux/4.18.0-240.15.1.el8_3.x86_64 exe/x86_64.rhel.8 prompt/off command/s3.ls","x-amz-date":"20210315T085400Z","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","authorization":"AWS4-HMAC-SHA256 Credential=6IUhji5FNL3rn6Eafrj0/20210315/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=870d2d0e726ea5433fb09013a854f8ece944613d28d7e46110761bf58f2ff31f"} Error: ENOENT: no such file or directory, mkdir '/nsfs/test-10'