High idle CPU usage

Greetings,

First of all, thanks for the great library!

I’ve encountered high idle CPU usage. This is especially noticeable when running multiple microservices (~15 instances) on the single AWS ec2 instance.

• loki4j version: 1.3.0
• logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>

    <if condition='isDefined("LOKI_URL")'>
        <then>
            <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
                <batchTimeoutMs>60000</batchTimeoutMs>
                <http>
                    <url>${LOKI_URL}</url>
                </http>
                <format>
                    <label>
                        <pattern>pod=${HOSTNAME},app=${APP_NAME},level=%level</pattern>
                    </label>
                    <message>
                        <pattern>l=%level c=%logger{20} t=%thread | %msg %ex</pattern>
                    </message>
                    <sortByTime>true</sortByTime>
                </format>
            </appender>

            <root level="INFO">
                <appender-ref ref="CONSOLE"/>
                <appender-ref ref="LOKI"/>
            </root>
        </then>
        <else>
            <root level="INFO">
                <appender-ref ref="CONSOLE"/>
            </root>
        </else>
    </if>

    <logger name="org.springframework.web" level="ERROR"/>
</configuration>

After disabling loki4j and enabling promtail idle load returned to normal

I’m not sure what can cause such behavior. Looking forward to your support

[nehaev]

Loki4j v1.3.1 was published, according to my measurements idle CPU usage should be lower. Feel free to re-open this issue if the fix didn’t work.

[nehaev]

Thanks for providing the profiling report on CPU usage! The root cause of this issue is this park timeout hardcoded to 1ms. This timeout defines the frequency that Loki4j checks its internal queues for new events to encode or send. Looks like 1ms cause too much thread context switching on idle.

There’s a tradeoff between encode/send latency and CPU usage, but I guess it should be safe to increase its value to 25ms as a temporary fix. In future versions I’ll consider to make a config property for this.