[BUG] Index patterns not loaded after creation of the index template for all indices pattern "*"

Describe the bug

If we load an index template for all indices “*” pattern (before dashboards pod starts and is up) with following template settings, indices patterns are not loaded anymore and saved objects are also not visible anymore and we get “Unable find saved objects” error on screen.

"template" : { "settings" : { "index" : { "routing" : { "allocation" : { "include" : { "role" : "data" } } }, "refresh_interval" : "30s" } } }

We checked that problem is caused by the fact that .kibana* system indices are getting the all indices index template applied and has template setting for routing allocation. ".kibana_1" : { "settings" : { "index" : { "routing" : { "allocation" : { "include" : { "role" : "data" } } }, "refresh_interval" : "30s", "number_of_shards" : "1", "auto_expand_replicas" : "0-1", "provided_name" : ".kibana_1", "creation_date" : "1668430185144", "number_of_replicas" : "1", "uuid" : "sPO3PtfcRC2DVn_ynIcl_A", "version" : { "created" : "136247827" } } } }

To Reproduce Steps to reproduce the behavior:

1. Go to Saved Objects

2. See error

3. But it seems, the saved object config is present there but not visible by default:

1. And if we create an index pattern, it doesn’t get saved.
2. We also noticed that, if we restart the dashboards pod, then dashboards gui starts to work fine!!

Expected behavior Opensearch dashboard functionality should not be impacted due to index allocation routing on .kibana* system indices.

OpenSearch Version 2.3.0

Dashboards Version 2.3.0

Plugins

No custom plugin enabled.

Screenshots

If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Browser and version [e.g. 22]

Additional context

Add any other context about the problem here.

[IanGabes]

Hi @ananzh, to your repro steps i would add:

• Have multi-tenancy enabled
• Select the private or a shared tenant

We do not notice the issue on the Global tenant. Thanks!

[joshuarrrr]

Yeah, @opensearch-project/transfer-request , I think this is more likely to be solved via https://github.com/opensearch-project/security-dashboards-plugin