Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
[BUG] repository-s3 plugin doesn't work outside of AWS
See original GitHub issueDescribe the bug Trying to add a snapshot repository fails on our bare-metal clusters. I am trying to add an S3-compatible endpoint as our data store, but I get the following stack trace:
[2022-05-26T23:49:05,642][WARN ][r.suppressed ] [esmaster-ams5-0000] path: /_snapshot/ams5_security, params: {repository=ams5_security}
org.opensearch.transport.RemoteTransportException: [graylog-ams5-0000][10.8.5.45:9300][cluster:admin/repository/put]
Caused by: org.opensearch.repositories.RepositoryVerificationException: [ams5_security] path [PROD-AC-AMS5] is not accessible on master node
at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1549) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.IOException: Unable to upload object [PROD-AC-AMS5/tests-CluPofPPTlG7tTGrUDgM2Q/master.dat] using a single upload
at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:379) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?]
at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.opensearch.common.io.stream.NotSerializableExceptionWrapper: sdk_client_exception: Failed to connect to service endpoint:
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[?:?]
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?]
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?]
at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?]
at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?]
at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:55) ~[?:?]
at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:294) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$18(S3BlobContainer.java:377) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:70) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:69) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:377) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?]
at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.io.IOException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?]
at java.net.Socket.connect(Socket.java:609) ~[?:?]
at sun.net.NetworkClient.doConnect(NetworkClient.java:177) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[?:?]
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:?]
at sun.net.www.http.HttpClient.New(HttpClient.java:341) ~[?:?]
at sun.net.www.http.HttpClient.New(HttpClient.java:362) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1232) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015) ~[?:?]
at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52) ~[?:?]
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80) ~[?:?]
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70) ~[?:?]
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75) ~[?:?]
at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[?:?]
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58) ~[?:?]
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46) ~[?:?]
at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112) ~[?:?]
at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68) ~[?:?]
at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166) ~[?:?]
at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:55) ~[?:?]
at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(S3Service.java:294) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5942) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1808) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1768) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$18(S3BlobContainer.java:377) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:70) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:69) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:377) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142) ~[?:?]
at org.opensearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:159) ~[?:?]
at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:305) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.3.1.jar:1.3.1]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.3.1.jar:1.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
It looks like the plugin is attempting to access EC2 resources for authentication, which is not possible. I have saved the proper credentials in the the local keystore.
To Reproduce Steps to reproduce the behavior:
- Setup a cluster, either on bare metal or any service provider aside from AWS.
- Save credentials in the keystore as specified in the documentation.
- Try and add an S3-compatible endpoint:
- See error
Expected behavior Adding a snapshot repository with the following settings should create the repository and allow me to start taking snapshots:
{
"type": "s3",
"settings": {
"bucket": "BUCKET_NAME",
"base_path": "DIRECTORY_NAME",
"endpoint": "s3.us-west-002.backblazeb2.com",
"region": "us-west-002"
}
}'
Plugins Please list all plugins currently enabled.
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-observability
opensearch-reports-scheduler
opensearch-security
opensearch-sql
repository-s3
Host/Environment (please complete the following information):
- OS: Debian 9.13
- Version : opensearch-1.3.1
Additional context This arrangement did work on Elasticsearch 7.10, with the Opendistro plugins.
Issue Analytics
- State:
- Created a year ago
- Comments:5
Top Results From Across the Web
Troubleshooting errors in AWS Glue
The AWS Glue GitHub repository contains additional troubleshooting guidance in AWS ... Error: Could not find S3 endpoint or NAT gateway for subnetId...
Read more >Troubleshoot the 403 Forbidden error when uploading files ...
Open the Amazon S3 console. · From the list of buckets, open the bucket you want to upload files to. · Choose the...
Read more >Troubleshooting import and export failures - FSx for Lustre
Amazon FSx was unable to import or export the file because it doesn't exist in the data repository. S3ObjectInUnsupportedTier. Amazon FSx successfully imported ......
Read more >Troubleshoot 403 Access Denied errors from Amazon S3
If the IAM user tries to modify the access control list (ACL) of an object, then the user gets an Access Denied error....
Read more >Troubleshooting common AWS CDK issues
When deploying my AWS CDK stack, I receive a NoSuchBucket error. Your AWS environment has not been bootstrapped, and so does not have...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
We finally made it work. I don’t remember how I initially created /etc/opensearch/opensearch.keystore but when I run /opt/opensearch/bin/opensearch-keystore create it puts the new keystore file in /opt/opensearch/config
So I removed all of the credentials from the keystore, made sure it was created in the /opt/opensearch subdirectory on every node, then copied that into /etc/opensearch, reloaded settings via the API, then I was actually able to create a repository and take a snapshot.
I have seen the similar issue in version : 1.2.4 Aug 16 14:51:30 ip-10-1-1-222 bash[9142]: opensearch.default(O): [2022-08-16T14:51:30,376][WARN ][o.o.r.s.S3BlobStore ] [ip-10-1-1-222] Expected request count to be tracked for request [PUT https://MYBUCKET.s3.amazonaws.com /Aug16/opensearch/tests-WW5UbIGRQu2tQe6dG0Pvsw/master.dat Headers: (Content-Length: 22, Content-Type: application/octet-stream, x-amz-acl: private, x-amz-storage-class: STANDARD, ) ] but found not count. Aug 16 14:51:30 ip-10-1-1-222 bash[9142]: opensearch.default(O): [2022-08-16T14:51:30,378][WARN ][r.suppressed ] [ip-10-1-1-222] path: /_snapshot/event-feed-service, params: {verify=true, repository=chef-automate-es6-event-feed-service} Aug 16 14:51:30 ip-10-1-1-222 bash[9142]: opensearch.default(O): org.opensearch.repositories.RepositoryVerificationException: [chef-automate-es6-event-feed-service] path [Aug16/opensearch/event-feed-service] is not accessible on master node
Is this fixed ?