[FEATURE REQUEST] Allow to disable Open Distro audit logs

Is your spike related to a problem or idea? Please describe. By default ELK stack doesn’t have any policy management for its default indices. There are security indices that are producing data all the time which ends with taking over all possible shards on ElasticSearch and ELK stack stops working.

Example issue from a customer:

{"agent":{"ephemeral_id":"cfcd68d5-ec76-424e-b2eb-1e89d1ac57c3","hostname":"lahcmonitor01d","id":"474b78b4-60ff-4fc5-9147-acce120f1dd2","name":"lahcmonitor01d","type":"filebeat","version":"7.9.2"},"ecs":{"version":"1.5.0"},"host":{"name":"lahcmonitor01d"},"input":{"type":"log"},"log":{"file":{"path":"/var/log/messages"},"offset":703865869},"message":"Feb 21 09:05:19 lahcmonitor01d performance-analyzer-agent-cli: 09:05:19.140 [pa-reader] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ReaderMetricsProcessor - Error reading file '/usr/share/elasticsearch/data/batch_metrics_enabled.conf': java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/batch_metrics_enabled.conf"}, Private:file.State{Id:"native::3195-64773", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000c24680), Source:"/var/log/messages", Offset:703866222, Timestamp:time.Time{wall:0xc07d018e2a7d621c, ext:8371919972062144, loc:(*time.Location)(0x419f7a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xc7b, Device:0xfd05}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"validation_exception","reason":"Validation Failed: 1: this action would add [6] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}

This indices are created by default (tbh same as filebeat*) so there should be some default policy to prevent this situation from happening.

Describe the outcome you’d like Check what kind of Policy management we can offer to keep this issue from happening in the form of additional issues. Note that these (might/should) need to cover the following:

• Elasticsearch for version 1.0.3

See comments below for approach taken.

What is the reason or source for the spike Internal team

Additional context None

---

DoD checklist

• Changelog
  • updated
  • not needed
• COMPONENTS.md
  • updated
  • not needed
• Schema
  • updated
  • not needed
• Backport tasks
  • created
  • not needed
• Documentation
  • added
  • updated
  • not needed
• Feature has automated tests
• Automated tests passed (QA pipelines)
  • apply
  • upgrade
  • backup/restore
• Idempotency tested
• All conversations in PR resolved
• Solution meets requirements and is done according to design doc
• Usage compliant with license

[seriva]

This PR is quite big already. I would recommend doing it in the separated one.

Could you create a new issue for that for 2.0.1, linking back to this issue and blocked by your opensearch one?

[atsikham]

According to documentation it’s almost the same for opensearch:

• https://opensearch.org/docs/latest/security-plugin/audit-logs/index/
• https://opensearch.org/docs/latest/im-plugin/ism/index/

As I see in PR #3093, we are going to enable audit logging by default again: