[BUG] Getting security exception due to access denied "java"

I am experiencing similar issues with GCS repositories as were discussed in https://github.com/opensearch-project/OpenSearch/issues/4269 for S3 repositories.

In 2.2.x the GCS snapshots were working fine running with Workload Identity in a GKE cluster, but starting with 2.3.0 i started seeing the same java security policy issues. In 2.3.0 I was still able to take snapshots even though it showed that all indices failed. With 2.4.0 I cannot even load the list of repositories anymore.

To Reproduce Steps to reproduce the behavior:

1. Setup GKE with workload identity and map KSA to GSA with bucket write/read/etc.
2. Deploy Opensearch with KSA and repository-gcs installed
3. Try to snapshot/load repositories/etc.
4. Errors

Note that all the missing permissions being complained about are configured in the plugins security policy file already by default.

trying to load repositories

[2022-11-22T03:14:17,406][WARN ][r.suppressed             ] [opensearch-cluster-main-0] path: /_snapshot/opencti-opensearch-backup-repository/_all, params: {ignore_unavailable=true, repository=opencti-opensearch-backup-repository, snapshot=_all}
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-managers-2][192.168.34.14:9300][cluster:admin/snapshot/get]
Caused by: org.opensearch.repositories.RepositoryException: [opencti-opensearch-backup-repository] Unexpected exception when loading repository data
        at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1734) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.4.0.jar:2.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.SecurityException: access denied ("java.net.SocketPermission" "metadata.google.internal:80" "connect,resolve")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
        at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:416) ~[?:?]
        at java.lang.SecurityManager.checkConnect(SecurityManager.java:919) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:619) ~[?:?]
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:279) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:384) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:406) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057) ~[?:?]
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:148) ~[?:?]
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[?:?]
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[?:?]
        at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:503) ~[?:?]
        at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472) ~[?:?]
        at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379) ~[?:?]
        at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356) ~[?:?]
        at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302) ~[?:?]
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:83) ~[?:?]
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:31) ~[?:?]
        at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:78) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:232) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:171) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:128) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:133) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:222) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:85) ~[?:?]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:1885) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1696) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.4.0.jar:2.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

trying to get info on specific snapshot

[2022-11-22T03:17:58,284][WARN ][r.suppressed             ] [opensearch-cluster-main-0] path: /_snapshot/opencti-opensearch-backup-repository/all, params: {pretty=true, repository=opencti-opensearch-backup-repository, snapshot=all}
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-managers-2][192.168.34.14:9300][cluster:admin/snapshot/get]
Caused by: org.opensearch.repositories.RepositoryException: [opencti-opensearch-backup-repository] Unexpected exception when loading repository data
        at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1734) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.4.0.jar:2.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.SecurityException: access denied ("java.net.SocketPermission" "metadata.google.internal:80" "connect,resolve")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
        at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:416) ~[?:?]
        at java.lang.SecurityManager.checkConnect(SecurityManager.java:919) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:619) ~[?:?]
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:279) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:384) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:406) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057) ~[?:?]
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:148) ~[?:?]
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[?:?]
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[?:?]
        at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:503) ~[?:?]
        at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472) ~[?:?]
        at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379) ~[?:?]
        at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356) ~[?:?]
        at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302) ~[?:?]
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:83) ~[?:?]
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:31) ~[?:?]
        at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:78) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:232) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:171) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:128) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:133) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:222) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:85) ~[?:?]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:1885) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1696) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.4.0.jar:2.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

Expected behavior Able to take, restore, etc. snapshots in combination with a GCS repository while running with Workload Identity (or similar).

Plugins

• repository-gcs

Screenshots

Host/Environment (please complete the following information):

• OS: docker.io/opensearchproject/opensearch:2.4.0
• Version 2.4.0

[max-frank]

Ok so it seems that permission java.net.SocketPermission "*", "connect,resolve"; is not necessary. I have tested with fresh clusters with the exact same deployment configuration and in both cases it worked fine. It seems the problem was that somehow the repository config on the original cluster were I experienced this issue got corrupted resulting in these issues.

Note sure how it got corrupted since the same repository config works fine now after deleting the repositories once and re-adding it with the original configuration. Only thing I can imagine that something went wrong during the cluster upgrade.

Though note that it seems the resolve permissions is detected as required during plugin install anyway even if its not explicitly part of the plugin-security.policy

Defaulted container "opensearch" out of: opensearch, init (init), init-sysctl (init)
-> Installing repository-gcs
-> Downloading repository-gcs from opensearch
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission setFactory
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve

Sorry for blowing up this issue the original issue is resolved though I have not been able to reproduce the cause yet.

[dblock]

Thanks for bringing this up and working through it with us @max-frank!