Filebeat not creating specified index

Chart version: 7.6.1

Kubernetes version: 1.4

Kubernetes provider: EKS

So I am trying to set up filebeat to use an index different to filebeat, below is my config map:

filebeat.inputs:
- type: docker
  containers.ids:
  - '*'
  processors:
  - add_kubernetes_metadata:
      in_cluster: true
  - add_cloud_metadata:

  fields:
    source: eu-west-1
    log_type: kubernetes
    environment_type: production
    environment: eu-west-1
  tags: ['kubernetes']

output.elasticsearch:
  index: "kubernetes"
  hosts: "https://xeu-west-1.aws.found.io:9243/"
  username: "elastic"
  password: "x"

setup.template.name: "kubernetes"
setup.template.pattern: "kubernetes-*"
setup.template.settings:
  setup.ilm.enabled: true
  setup.ilm.rollover_alias: "kubernetes"
  setup.ilm.pattern: "{now/d}-000001"
  setup.ilm.policy_name: "Standard-Index"

however, when the pod starts it appears to treat ILM as if it’s set to auto.

2020-03-23T09:50:59.310Z	INFO	[index-management]	idxmgmt/std.go:258	Auto ILM enable success.
2020-03-23T09:50:59.315Z	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
2020-03-23T09:50:59.315Z	INFO	[index-management]	idxmgmt/std.go:271	ILM policy successfully loaded.
2020-03-23T09:50:59.315Z	INFO	[index-management]	idxmgmt/std.go:410	Set setup.template.name to '{filebeat-7.6.1 {now/d}-000001}' as ILM is enabled.
2020-03-23T09:50:59.315Z	INFO	[index-management]	idxmgmt/std.go:415	Set setup.template.pattern to 'filebeat-7.6.1-*' as ILM is enabled.

I am using image and chart version 7.6.1 there are zero errors or warnings on fiilebeat startup.

It’s definitely reading the config as the logs are reaching our ES cluster in the filebeat index. How do I change this index?

[fatmcgav]

You are 100% correct, my apologies

No worries. Glad I could get you working…

[fatmcgav]

@runtman it looks like the setup.ilm.* keys are still nested under the setup.template.settongs: key.

They shouldn’t be…

So the config should look like:

setup.template.name: "kubernetes"
setup.template.pattern: "kubernetes-*"
setup.ilm.enabled: true
setup.ilm.rollover_alias: "kubernetes"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "Standard-Index"