Upgrading 1.2.0 to 1.3.0 - logging pods fail with error="'ingest_api_tenant' parameter is required"

What happened: Installed 1.3.0 the same way I installed 1.2.0 and previous releases: helm install --name splunk-connect -f ${WELLTOK_ENVIRONMENT}-values.yml https://github.com/splunk/splunk-connect-for-kubernetes/releases/download/1.3.0/splunk-connect-for-kubernetes-1.3.0.tgz Logging pods fail with error:

fluentd -c /fluentd/etc/fluent.conf
2019-12-18 17:29:11 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
/usr/local/bundle/gems/activesupport-3.2.22.5/lib/active_support/core_ext/object/duplicable.rb:111: warning: BigDecimal.new is deprecated; use BigDecimal() method instead.
2019-12-18 17:29:12 +0000 [error]: config error in:
<match **>
  @type splunk_ingest_api
  host "ip-10-52-46-109.ec2.internal"
  source_key "source"
  sourcetype_key "sourcetype"
  <fields>
    pod
    namespace
    container_name
    container_id
    cluster_name
  </fields>
  <buffer>
    @type memory
    chunk_limit_records 100000
    chunk_limit_size 200m
    flush_interval 5s
    flush_thread_count 1
    overflow_action block
    retry_max_times 3
    total_limit_size 600m
  </buffer>
  <format monitor_agent>
    @type json
  </format>
  <format>
    @type single_value
    message_key log
    add_newline false
  </format>
</match>

2019-12-18 17:29:12 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="'ingest_api_tenant' parameter is required"

What you expected to happen: Splunk logging How to reproduce it (as minimally and precisely as possible): Install above. Anything else we need to know?: values file:

# This is an example of using insecure configurations
global:
  splunk:
    hec:
      token: <redacted>
      host: <redacted>
      port: 8088
      indexName: kubernetes-nonprod-logs
      # connection to splunk is insecure
      insecureSSL: true
  kubernetes:
    clusterName: tester1

splunk-kubernetes-logging:
  logs:
    kube-apiserver-audit:
      from:
        file:
          path: /var/log/kube-apiserver-tester1-audit.log
  kubernetes:
    clusterName: tester1

splunk-kubernetes-objects:
  # RBAC is disabled
  rbac:
    create: true
  splunk:
    hec:
      indexName: kubernetes-nonprod-objects
  kubernetes:
    # connection to kubernetes is insecure
    insecureSSL: true
    clusterName: tester1

splunk-kubernetes-metrics:
  # RBAC is disabled
  rbac:
    create: true
  # do not create service account
  serviceAccount:
    create: true
  splunk:
    hec:
      indexName: kubernetes-nonprod-metrics
  kubernetes:
    clusterName: tester1

Environment:

• Kubernetes version (use kubectl version): » kubectl version

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:42:56Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:34:17Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

• Splunk version:

Splunk Enterprise
Version:
7.3.2
Build:
c60db69f8e32
Server:
wtsh1

• Others:

[dmcnaught]

ok, adding it to splunk logging too worked:

splunk-kubernetes-logging:
  logs:
    kube-apiserver-audit:
      from:
        file:
          path: /var/log/kube-apiserver-tester-audit.log
  kubernetes:
    clusterName: tester  
  splunk:
    hec:
      token: <redacted>
      host: <redacted>
      port: 8088
      indexName: kubernetes-nonprod-logs
      # connection to splunk is insecure
      insecureSSL: true

[dmcnaught]

There is still an issue here @chaitanyaphalak - we shouldn’t have to define splunk.hec values in global and logging sections of a template.