Fine-grained logging configuration and consistency

One of the most important considerations in running production services is logging. It’s common practice to aggregate Kubernetes container logs (either the entire container log from the host system or specific files via a sidecar container) for analysis and search with a tool like Splunk or ElasticSearch. Ambassador’s logging is currently pretty sad: only logs to stdout/stderr, different components have different log formats, and some processes log without a datestamp altogether. All of this makes it more difficult than it should be to index Ambassador logs.

Ambassador should add configuration options for logging:

• Envoy access log format (#701) https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log
• Log format for Python components (hot-restarter, kubewatch, diagd) using logging module (#441) https://docs.python.org/3/library/logging.html
• Prefix for all logs in entrypoint.sh (including date(1) formatting for timestamps)
• Log destination for all components, should allow writing each component’s logs to separate files (at specified paths) in addition to stdout/stderr
• Ideal: a new LoggingService manifest for configuring direct log forwarding to Splunk HEC/ElasticSearch/etc.

Ambassador’s default configuration should log everything with the same format (at very least with a consistent timestamp prefix, including time zone).

I’ve attempted to use the sidecar method with a Splunk forwarder container. I had to write my own Dockerfile that depends on Ambassador and adds a custom entrypoint to tee the logs into files:

Dockerfile

ARG VERSION
FROM quay.io/datawire/ambassador:${VERSION}

RUN apk --no-cache add bash

WORKDIR /ambassador
COPY my_entrypoint.sh .
RUN chmod 755 my_entrypoint.sh

ENTRYPOINT [ "./my_entrypoint.sh" ]

my_entrypoint.sh

#!/bin/bash

mkdir -p /logs

exec > >(tee -a /logs/ambassador_stdout.log)
exec 2> >(tee -a /logs/ambassador_stderr.log)

# variants with added timestamps, but this causes double timestamps on some logs???
# exec > >(awk '{ print strftime("%Y-%m-%d %H:%M:%S %z"), $0; fflush(); }' | tee -a /logs/ambassador_stdout.log)
# exec 2> >(awk '{ print strftime("%Y-%m-%d %H:%M:%S %z"), $0; fflush(); }' | tee -a /logs/ambassador_stderr.log)

exec ./entrypoint.sh

(Installing Bash is required to get >() process redirection)

[HT154]

nooope

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.