Optimize fluentd buffers and guidance

What would you like to be added:

The Fluentd buffers should be optimized and guidance and simple test results should be shared.

https://github.com/splunk/splunk-connect-for-kubernetes/blob/e680eb1d6fe53c0346ad66ddb22b3d8ee0703226/helm-chart/splunk-connect-for-kubernetes/values.yaml#L635-L643

  buffer:
    "@type": memory
    total_limit_size: 600m
    chunk_limit_size: 10m
    chunk_limit_records: 10000
    flush_interval: 3s
    flush_thread_count: 1
    overflow_action: block
    retry_max_times: 5
    retry_type: periodic

Why is this needed:

chunk_limit_size has tested best at 10-20MB in high volume environments when customers have been doing scale/tress tests, plus we want to keep thruput high and chunk size low! We should try never to be sending a payload that big.

retry_type keeps us away from fluentd’s exponential backoff timer, which can cause fun lag issues. We should lean to always firing out data as fast as we can. dont want to be holding buffers. Saw an instance where after failed attempts fluentd would wait 24 hours to send

[vinzent]

having set the retry_max_interval to something reasonable should workaround the silly 1d retry wait intervals seen with default config and also cope with very short issues better than just a periodic 60s wait time.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.