Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Log messages not pushed to Logstash

See original GitHub issue

We using logstash-logback-encoder version 4.11 with Spring Boot 1.5.6-RELEASE, and jHipster, deployed in Cloud Foundry.

We have configured the logger programmatically. While the logs are seen on the console we notice that many are simply not being TCP-ed to the Logstash server. Moreover, there is no error or message indicating that there have been drops although the TCP connection itself looks perfectly healthy. In fact, we have a heartbeat, which is working just fine.

Logger configuration:

//Additional constants...
  private static final String ASYNC_APPENDER_NAME = "ASYNC";
    private static final String CONSOLE_APPENDER_NAME = "CONSOLE";
    private static final Integer ASYNC_QUEUE_SIZE = 1000;
    private static final Integer ASYNC_FLUSH_SIZE = 2000;
    private static final Integer MAX_HISTORY = 30;
    private static final Integer DISCARDING_THRESHOLD = 0;
public static void configure() {
        LoggerContext loggerContext = loggerContext();

        StatusManager statusManager = loggerContext.getStatusManager();
        if (statusManager != null) {
            statusManager.add(new InfoStatus("Setting Logger configuration from " + LoggerConfiguration.class.getName(), loggerContext));
            statusManager.add(new OnErrorConsoleStatusListener());
        }

        loggerContext.addListener(levelChangePropogator());
        setCustomLogLevels(loggerContext);

        StatusPrinter.print(statusManager);
        StatusPrinter.print(loggerContext);
        StatusPrinter.printInCaseOfErrorsOrWarnings(loggerContext);
    }

    @Bean
    public static LoggerContext loggerContext() {
        return (LoggerContext) LoggerFactory.getILoggerFactory();
    }

    @Bean
    public static Logger rootLogger(LoggerContext loggerContext) {
        Logger rootLogger = loggerContext.getLogger(Logger.ROOT_LOGGER_NAME);
        rootLogger.setLevel(Level.INFO);
        rootLogger.addAppender(consoleAppender(loggerContext, rootLogger));
        rootLogger.addAppender(asyncAppender(loggerContext, tcpSocketAppender(loggerContext, rootLogger)));
        rootLogger.addAppender(rollingFileAppender(loggerContext));
        return rootLogger;
    }

    @Bean
    protected static LevelChangePropagator levelChangePropogator() {
        LevelChangePropagator levelChangePropagator = new LevelChangePropagator();
        levelChangePropagator.setResetJUL(true);
        return levelChangePropagator;
    }

    @Bean
    @Primary
    protected static LogstashTcpSocketAppender tcpSocketAppender(LoggerContext loggerContext, Logger rootLogger) {
        LogstashTcpSocketAppender logstashTcpSocketAppender = new LogstashTcpSocketAppender();
        logstashTcpSocketAppender.setName(TCP_APPENDER_NAME);
        logstashTcpSocketAppender.setContext(loggerContext);
        logstashTcpSocketAppender.addDestination(TCP_DESTINATION);
        logstashTcpSocketAppender.setWriteBufferSize(TCP_WRITE_BUFFER_SIZE);
        logstashTcpSocketAppender.setEncoder(logstashEncoder(loggerContext, rootLogger));
        logstashTcpSocketAppender.setKeepAliveDuration(Duration.buildByDays(100));

        return logstashTcpSocketAppender;
    }

    @Bean
    protected static LogstashEncoder logstashEncoder(LoggerContext loggerContext, Logger rootLogger) {
        LogstashEncoder encoder = new LogstashEncoder();
        encoder.setContext(loggerContext);
        encoder.setIncludeContext(true);
        encoder.setCustomFields(setEnvironmentAttributes());
        encoder.start();

        return encoder;
    }

    @Bean
    protected static ConsoleAppender<ILoggingEvent> consoleAppender(LoggerContext loggerContext, Logger rootLogger) {
        ConsoleAppender<ILoggingEvent> consoleAppender = new ConsoleAppender<>();
        consoleAppender.setContext(loggerContext);
        consoleAppender.setName(CONSOLE_APPENDER_NAME);
        consoleAppender.setEncoder(patternLayoutEncoder(loggerContext));
        consoleAppender.start();
        return consoleAppender;
    }

The logstash input configuration is below:

tcp {
       port => 9202
       type => "some_type"
       tags => [ "some_tag" ]
       codec => json_lines
   }

Any thoughts why this could be occurring? Importantly, how best to debug such an issue? Relevant code snippets are included - please let me know if additional information is required. Any inputs will be of great help.

Thanks!

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:8

github_iconTop GitHub Comments

1reaction
saketb01commented, Oct 10, 2017

Hi @philsttr thank you for your inputs and valuable suggestions.

  1. TCP_WRITE_BUFFER_SIZE: Tried different values here, including letting the default be. This does not seem to matter at this point.
  2. Updated logger configuration in entirety is below - the start() is being called explicitly
  3. setCustomLevels() is explicitly setting log levels for libraries, and also the root package for the service in question

Starting the nc command has helped narrow down the problem in the sense that logs are “selectively” being TCP-ed. Specifically, all the library logs (spring, etc.) are going through fine but the application-level are not.

import java.nio.charset.Charset;
import java.nio.charset.Charset;

import javax.annotation.PostConstruct;

import org.apache.commons.lang3.StringUtils;
import org.json.JSONException;
import org.json.JSONObject;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.DependsOn;

import ch.qos.logback.classic.AsyncAppender;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.LoggerContext;
import ch.qos.logback.classic.encoder.PatternLayoutEncoder;
import ch.qos.logback.classic.filter.ThresholdFilter;
import ch.qos.logback.classic.jul.LevelChangePropagator;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.ConsoleAppender;
import ch.qos.logback.core.rolling.RollingFileAppender;
import ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy;
import ch.qos.logback.core.rolling.TimeBasedRollingPolicy;
import ch.qos.logback.core.status.InfoStatus;
import ch.qos.logback.core.status.OnErrorConsoleStatusListener;
import ch.qos.logback.core.status.StatusManager;
import ch.qos.logback.core.util.FileSize;
import ch.qos.logback.core.util.StatusPrinter;
import net.logstash.logback.appender.LogstashTcpSocketAppender;
import net.logstash.logback.encoder.LogstashEncoder;

@Configuration
public class LoggerConfiguration {

    private static final String LOG_LEVEL_PATTERN = System.getProperty("LOG_LEVEL_PATTERN") == null ? "-%5p" : System.getProperty("LOG_LEVEL_PATTERN");
    private static final String LOG_EXCEPTION_CONVERSION_WORD = System.getProperty("LOG_EXCEPTION_CONVERSION_WORD") == null ? "-%wEx" : System.getProperty("LOG_EXCEPTION_CONVERSION_WORD");
    private static final String HOME = System.getProperty("user.home") == null ? "/app/logs/" : System.getProperty("user.home");

    private static final String PATTERN = "%d{yyyy-MM-dd HH:mm:ss.SSS}%clr(" + LOG_LEVEL_PATTERN + ") %clr([%15.15t]){faint}%clr(%-40.40logger{39}){cyan} %clr(:){faint}%m%n" + LOG_EXCEPTION_CONVERSION_WORD;

    private static final String ROLLING_FILE_APPENDER_NAME = "ROLLING";
    private static final String ROLLING_FILENAME = HOME + "/device_inquiry.log";
    private static final String ROLLING_FILENAME_PATTERN = HOME + "/archived/device_inquiry.%d{yyyy-MM-dd}.%s.log";
    private static final String MAX_FILESIZE = "10MB";
    private static final String CHARSET = "UTF-8";

    private static final String TCP_DESTINATION = "localhost:8081"; //for nc -lk 8081
    private static final String TCP_APPENDER_NAME = "stash";
    private static final Integer TCP_WRITE_BUFFER_SIZE = 81920; // 32_768;
    
    private static final String ASYNC_APPENDER_NAME = "ASYNC";
    private static final String CONSOLE_APPENDER_NAME = "CONSOLE";

    private static final Integer ASYNC_QUEUE_SIZE = 1000;
    private static final Integer ASYNC_FLUSH_SIZE = 2000;
    private static final Integer MAX_HISTORY = 30;
    private static final Integer DISCARDING_THRESHOLD = 0;

    private static final org.slf4j.Logger LOGGER = LoggerFactory.getLogger(LoggerConfiguration.class);

    // Environment, datacenter and host are grabbed from Cloud Environment
    private String environment;
    private String datacenter;
    private String host;

    @Bean("loggerContext")
    public LoggerContext loggerContext() {
        LOGGER.info(" configure loggerContext");
        return (LoggerContext) LoggerFactory.getILoggerFactory();
    }

    @PostConstruct
    public void configureLoggerContext() {
        LOGGER.info("@PostConstruct  configure LoggerContext");
        LoggerContext loggerContext = loggerContext();
        //rootLogger(loggerContext);
        //tcpSocketAppender(loggerContext);
        //consoleAppender(loggerContext);
        //asyncAppender(loggerContext, tcpSocketAppender(loggerContext));

        onErrorStatusListener(loggerContext);
        loggerContext.addListener(levelChangePropogator(loggerContext));
        setCustomLogLevels(loggerContext);

        StatusManager statusManager = loggerContext.getStatusManager();
        if (statusManager != null) {
            statusManager.add(new InfoStatus(" Setting Logger configuration from " + LoggerConfiguration.class.getName(), loggerContext));
        } else {
            LOGGER.error("Status Manager for logger does not exist");
        }
        StatusPrinter.print(statusManager);
        StatusPrinter.print(loggerContext);
        StatusPrinter.printInCaseOfErrorsOrWarnings(loggerContext);
        LOGGER.info(" configure - created logger");
    }

    @Bean
    @DependsOn("loggerContext")
    public OnErrorConsoleStatusListener onErrorStatusListener(LoggerContext loggerContext) {
        LOGGER.info(" Set up consoleStatusListener");
        OnErrorConsoleStatusListener statusListener = new OnErrorConsoleStatusListener();
        statusListener.setContext(loggerContext);
        loggerContext.getStatusManager().add(statusListener);
        statusListener.start();
        return statusListener;
    }

    @Bean
    @DependsOn("loggerContext")
    public Logger rootLogger(LoggerContext loggerContext) {
        LOGGER.info(" created rootLogger");
        Logger rootLogger = loggerContext.getLogger(Logger.ROOT_LOGGER_NAME);
        rootLogger.setLevel(Level.ALL);
        rootLogger.warn("WARN");
        return rootLogger;
    }

    @Bean
    @DependsOn("loggerContext")
    public LevelChangePropagator levelChangePropogator(LoggerContext loggerContext) {
        LOGGER.info(" created levelChangePropogator");
        LevelChangePropagator levelChangePropagator = new LevelChangePropagator();
        levelChangePropagator.setResetJUL(true);
        levelChangePropagator.setContext(loggerContext);
        return levelChangePropagator;
    }

    @Bean
    @DependsOn("rootLogger")
    public LogstashTcpSocketAppender tcpSocketAppender(LoggerContext loggerContext) {
        LOGGER.info(" created tcpSocketAppender");
        LogstashTcpSocketAppender logstashTcpSocketAppender = new LogstashTcpSocketAppender();
        logstashTcpSocketAppender.setName(TCP_APPENDER_NAME);
        logstashTcpSocketAppender.setContext(loggerContext);
        logstashTcpSocketAppender.addDestination(TCP_DESTINATION);
        logstashTcpSocketAppender.setWriteBufferSize(TCP_WRITE_BUFFER_SIZE);
        logstashTcpSocketAppender.setEncoder(logstashEncoder(loggerContext));
        logstashTcpSocketAppender.start();
        loggerContext.getLogger(Logger.ROOT_LOGGER_NAME).addAppender(logstashTcpSocketAppender);
        LOGGER.info(" done creating tcpSocketAppender");
        return logstashTcpSocketAppender;
    }

    @Bean
    @DependsOn("rootLogger")
    public LogstashEncoder logstashEncoder(LoggerContext loggerContext) {
        LOGGER.info(" created logstashEncoder");
        LogstashEncoder encoder = new LogstashEncoder();
        encoder.setContext(loggerContext);
        encoder.setIncludeContext(true);
        encoder.setCustomFields(setEnvironmentAttributes());
        encoder.start();
        return encoder;
    }

    @Bean
    @DependsOn("rootLogger")
    public ConsoleAppender<ILoggingEvent> consoleAppender(LoggerContext loggerContext) {
        LOGGER.info(" created consoleAppender");
        ConsoleAppender<ILoggingEvent> consoleAppender = new ConsoleAppender<>();
        consoleAppender.setEncoder(logstashEncoder(loggerContext));
        consoleAppender.setContext(loggerContext);
        consoleAppender.setName(CONSOLE_APPENDER_NAME);
        consoleAppender.setEncoder(patternLayoutEncoder(loggerContext));
        loggerContext.getLogger(Logger.ROOT_LOGGER_NAME).addAppender(consoleAppender);
        consoleAppender.start();
        return consoleAppender;
    }

    @Bean
    @DependsOn("rootLogger")
    public RollingFileAppender<ILoggingEvent> rollingFileAppender(LoggerContext loggerContext) {
        LOGGER.info(" created rollingFileAppender");
        RollingFileAppender<ILoggingEvent> rollingFileAppender = new RollingFileAppender<>();
        rollingFileAppender.setContext(loggerContext);
        rollingFileAppender.setName(ROLLING_FILE_APPENDER_NAME);
        rollingFileAppender.setFile(ROLLING_FILENAME);
        rollingFileAppender.setEncoder(patternLayoutEncoder(loggerContext));
        rollingFileAppender.setAppend(true);
        rollingFileAppender.setRollingPolicy(timeBasedRollingPolicy(loggerContext, rollingFileAppender));
        rollingFileAppender.setTriggeringPolicy(sizeBasedTriggeringPolicy(loggerContext));
        rollingFileAppender.addFilter(thresholdFilter(loggerContext));
        rollingFileAppender.start();
        return rollingFileAppender;
    }

    @Bean
    @DependsOn("rootLogger")
    public TimeBasedRollingPolicy<ILoggingEvent> timeBasedRollingPolicy(LoggerContext loggerContext, RollingFileAppender<ILoggingEvent> rollingFileAppender) {
        LOGGER.info(" created timeBasedRollingPolicy");
        TimeBasedRollingPolicy<ILoggingEvent> rollingPolicy = new TimeBasedRollingPolicy<>();
        rollingPolicy.setContext(loggerContext);
        rollingPolicy.setParent(rollingFileAppender);
        rollingPolicy.setFileNamePattern(ROLLING_FILENAME_PATTERN);
        rollingPolicy.setMaxHistory(MAX_HISTORY);
        rollingPolicy.start();
        return rollingPolicy;
    }

    @Bean
    @DependsOn("rootLogger")
    public SizeBasedTriggeringPolicy<ILoggingEvent> sizeBasedTriggeringPolicy(LoggerContext loggerContext) {
        LOGGER.info(" created SizeBasedTriggeringPolicy");
        SizeBasedTriggeringPolicy<ILoggingEvent> triggeringPolicy = new SizeBasedTriggeringPolicy<>();
        triggeringPolicy.setContext(loggerContext);
        triggeringPolicy.setMaxFileSize(FileSize.valueOf(MAX_FILESIZE));
        triggeringPolicy.start();
        return triggeringPolicy;
    }

    @Bean
    @DependsOn("rootLogger")
    public ThresholdFilter thresholdFilter(LoggerContext context) {
        LOGGER.info(" created thresholdFilter");
        ThresholdFilter thresholdFilter = new ThresholdFilter();
        thresholdFilter.setName("file-thresholdFilter");
        thresholdFilter.setContext(context);
        thresholdFilter.setLevel(Level.WARN.toString());
        thresholdFilter.start();
        return thresholdFilter;
    }

    @Bean
    @DependsOn("rootLogger")
    public AsyncAppender asyncAppender(LoggerContext loggerContext, LogstashTcpSocketAppender appender) {
        LOGGER.info(" created asyncAppender");
        AsyncAppender asyncAppender = new AsyncAppender();
        asyncAppender.setName(ASYNC_APPENDER_NAME);
        asyncAppender.setContext(loggerContext);
        asyncAppender.setDiscardingThreshold(DISCARDING_THRESHOLD);
        asyncAppender.setQueueSize(ASYNC_QUEUE_SIZE);
        asyncAppender.setMaxFlushTime(ASYNC_FLUSH_SIZE);
        asyncAppender.addAppender(appender);
        loggerContext.getLogger(Logger.ROOT_LOGGER_NAME).addAppender(rollingFileAppender(loggerContext));
        asyncAppender.start();
        return asyncAppender;
    }

    @Bean
    @DependsOn("rootLogger")
    public PatternLayoutEncoder patternLayoutEncoder(LoggerContext loggerContext) {
        LOGGER.info(" created patternLayoutEncoder");
        PatternLayoutEncoder encoder = new PatternLayoutEncoder();
        encoder.setContext(loggerContext);
        encoder.setPattern(PATTERN);
        encoder.setCharset(Charset.forName(CHARSET));
        encoder.start();
        return encoder;
    }

    private String setEnvironmentAttributes() {
        String envAttributesAsJsonString = "";
        try {
            envAttributesAsJsonString = new JSONObject().put("host", host).put("environment", environment).put("datacenter", datacenter).toString();
        } catch (JSONException jex) {
            jex.printStackTrace();
            LOGGER.error("setEnvironmentAttributes method could not set the host, environment and datacenter");
        }
        return envAttributesAsJsonString;
    }

    protected static void setCustomLogLevels(LoggerContext loggerContext) {
        loggerContext.getLogger("javax.management.modelmbean").setLevel(Level.WARN);
        loggerContext.getLogger("com.application.package.root").setLevel(Level.INFO);
        loggerContext.getLogger("org.apache.http.wire").setLevel(Level.WARN);
        loggerContext.getLogger("javax.activation").setLevel(Level.WARN);
        loggerContext.getLogger("javax.mail").setLevel(Level.WARN);
        loggerContext.getLogger("javax.xml.bind").setLevel(Level.WARN);
        loggerContext.getLogger("ch.qos.logback").setLevel(Level.WARN);
        loggerContext.getLogger("com.codahale.metrics").setLevel(Level.WARN);
        loggerContext.getLogger("com.netflix").setLevel(Level.WARN);
        loggerContext.getLogger("com.netflix.discovery").setLevel(Level.WARN);
        loggerContext.getLogger("com.ryantenney").setLevel(Level.WARN);
        loggerContext.getLogger("com.sun").setLevel(Level.WARN);
        loggerContext.getLogger("com.zaxxer").setLevel(Level.WARN);
        loggerContext.getLogger("org.apache").setLevel(Level.WARN);
        loggerContext.getLogger("org.apache.catalina.startup.DigesterFactory").setLevel(Level.OFF);
        loggerContext.getLogger("org.bson").setLevel(Level.WARN);
        loggerContext.getLogger("org.I0Itec.zkclient").setLevel(Level.WARN);
        loggerContext.getLogger("org.hibernate.validator").setLevel(Level.WARN);
        loggerContext.getLogger("org.hibernate").setLevel(Level.WARN);
        loggerContext.getLogger("org.hibernate.ejb.HibernatePersistence").setLevel(Level.WARN);
        loggerContext.getLogger("org.springframework").setLevel(Level.WARN);
        loggerContext.getLogger("org.springframework.web").setLevel(Level.WARN);
        loggerContext.getLogger("org.springframework.security").setLevel(Level.WARN);
        loggerContext.getLogger("org.springframework.cache").setLevel(Level.WARN);
        loggerContext.getLogger("org.springframework.integration").setLevel(Level.WARN);
        loggerContext.getLogger("org.springframework.cloud").setLevel(Level.WARN);
        loggerContext.getLogger("org.springframework.jmx").setLevel(Level.WARN);
        loggerContext.getLogger("org.springframework.boot.context").setLevel(Level.WARN);
        loggerContext.getLogger("org.thymeleaf").setLevel(Level.WARN);
        loggerContext.getLogger("org.xnio").setLevel(Level.WARN);
        loggerContext.getLogger("springfox").setLevel(Level.WARN);
        loggerContext.getLogger("sun.rmi").setLevel(Level.WARN);
        loggerContext.getLogger("sun.net").setLevel(Level.WARN);
        loggerContext.getLogger("liquibase").setLevel(Level.WARN);
        loggerContext.getLogger("sun.rmi.transport").setLevel(Level.WARN);
    }

    @Value("${vcap.application.space_name:}")
    public void setRuntimeEnvironment(String runTimeEnvironment) {
        LOGGER.info(" created logger runTimeEnvironment set to: {}", runTimeEnvironment);
        environment = runTimeEnvironment;
    }

    @Value("${vcap.application.application_uris[0]:}")
    public void setRuntimeDataCenter(String runTimedataCenter) {
        LOGGER.info(" created logger runTimedataCenter set to: {}", runTimedataCenter);
        datacenter = (StringUtils.isNotEmpty(runTimedataCenter) && runTimedataCenter.contains(".")) ? runTimedataCenter.split("\\.")[1] : "";
    }

    @Value("${vcap.application.instance_id:}" + "-" + "${vcap.application.name:}")
    public void setRuntimeHost(String runTimehost) {
        LOGGER.info(" created logger runTimehost set to: {}", runTimehost);
        host = runTimehost;
    }

}
0reactions
philsttrcommented, Dec 7, 2018

Closing due to inactivity

Read more comments on GitHub >

github_iconTop Results From Across the Web

Logstash logs are read but are not pushed to elasticsearch
Logstash logs are read but are not pushed to elasticsearch ... Elasticsearch responds with "acknowledged": true and I can see the template being...
Read more >
Logstash logs are read but are not pushed to elasticsearch
Now starting logstash with debug log level i see the input logs being read but not sent to elasticsearch, although the index is...
Read more >
Log messages not pushed to Logstash · Issue #252 - GitHub
We using logstash-logback-encoder version 4.11 with Spring Boot 1.5.6-RELEASE, and jHipster, deployed in Cloud Foundry.
Read more >
Logstash Tutorial: How to Get Started Shipping Logs - Logz.io
This introductory Logstash tutorial will get you started with everything from Logstash input and output plugins to using syslog and other apps like...
Read more >
Reporting logging data to Logstash in an ELK stack - IBM
The logging data that is sent to Logstash contains information about the events that are issued by the integration server process; events that ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

The message attribute in the JSON log has backslash, new line and special charecters

Next page

LogstashEncoder not compatible with Logback 1.2.x