Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Ability to add log arguments per Log.Entry basis without using formatted message

See original GitHub issue

Issue Description

We use LogStage for structured logging. For that we use JSON console sink, its output is parsed by Filebeats and then redirected to Graylog. All works fine except one thing: we want to be able to add additional diagnostic key-value arguments on a per Log.Entry basis, using LogIO facility, but in a such way that arguments don’t have place in rendered log message.

For example, if our application will try to export some domain entity to external system and failed to do that, we may want to log entity’s ID in message and attach formatted entity attributes (which may be quite complex in structure) in a form of JSON for diagnostic and debug purposes. Right now we must explicitly include formatted entity attributes in log message:

def importEntity(entity: Entity): IO[Unit] = /* ... */

importEntity(someEntity).handleErrorWith {
  case NonFatal(error) => LogIO[IO].log(Level.Error)(
    s"Failed to import entity ${ -> "ID"}: $error. ${entity.asJson.pretty(Printer.spaces2) -> "entityAsJSON"}").void

It works, but such log message will be rather ugly and log entry will duplicate content of entityAsJSON twice: in the log message and in rendered arguments. It also means additional overhead for network traffic and processing of incoming log entries by Graylog.

Do you think it’s meaningful use case?

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

nartamonovcommented, Apr 30, 2019

Sorry for late answer, thank you for suggestions! I decided to follow second advice from @kaishh and wrote simple wrapper around LogIO:

trait MonadLog[F[_]] {
  val monad: Monad[F]

  def log[E: ToLogMsg](level: Log.Level)(event: E)(implicit pos: CodePositionMaterializer): F[Unit]

object MonadLog {
  def apply[F[_]](implicit ev: MonadLog[F]): MonadLog[F] = ev

  def fromLogger[F[_]: SyncSafe: Monad](logger: IzLogger): MonadLog[F] = {
    implicit val logIO: LogIO[F] = LogIO.fromLogger[F](logger)
    new MonadLogForLogIO[F]

class MonadLogForLogIO[F[_]: LogIO: Monad] extends MonadLog[F] {
  val monad: Monad[F] = Monad[F]

  def log[E: ToLogMsg](level: Log.Level)(event: E)(implicit pos: CodePositionMaterializer): F[Unit] = {
    val msg = ToLogMsg[E].toLogMsg(event)
    val ctx = buildCustomContext(msg.context, eventType = event.getClass.getSimpleName)
    for {
      entry    <- LogIO[F].createEntry(level,msg.message)
      ctxEntry = entry.addCustomContext(ctx)
      _        <- LogIO[F].log(ctxEntry)
    } yield ()

  private def buildCustomContext(context: Map[String,Any], eventType: String): CustomContext = {
    val args = { case (k,v) => LogArg(Seq(k), v, hidden = false) }.toList
    val eventTypeArg = LogArg(Seq("event_type"), eventType, hidden = false)
    CustomContext(args :+ eventTypeArg)

trait ToLogMsg[-E] {
  def toLogMsg(e: E): LogMsg

object ToLogMsg {
  def apply[E](implicit ev: ToLogMsg[E]): ToLogMsg[E] = ev

case class LogMsg(message: Log.Message, context: Map[String,Any] = Map.empty)

It is typeclass for monad with logging capability (MTL style), which treats log messages not as simple strings but as ADTs - I prefer this approach in my current project. And it has simpler interface (1 method versus 4 in LogIO), so it’s easy to write fake implementation for testing purposes - you need to implement single method.

Now I can use that typeclass this way:

sealed trait AppLogEvent
// ...
case class FailedToImportEntity(entity: Entity, error: Throwable) extends LogEvent
// ...

object LogEvent extends DiagnosticEncoders {
  implicit val toLogMsg: ToLogMsg[LogEvent] = {
    // ...
    case FailedToImportEntity(entity, error) =>
      LogMsg(s"Failed to import entity ${ -> "entityId"}: $error.",
        context = Map("entity_data" -> stringify(entity)))
    // ...

def importEntity(entity: Entity): IO[Unit] = /* ... */

implicit val monadLog: MonadLog[IO] = MonadLog.fromLogger[IO](izLogger);
import monadLog._

importEntity(someEntity).handleErrorWith {
  case NonFatal(error) => log(Level.Error)(FailedToImportEntity(entity, error)).void

It seems all works fine, thank you! 😃

pshirshovcommented, Apr 30, 2019

Please note that this API is already available in latest snapshot. Also I would recommend you to join our gitter channel to follow our announcements.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Logging Cookbook — Python 3.11.1 documentation
Sometimes you want logging output to contain contextual information in addition to the parameters passed to the logging call. For example, in a...
Read more >
Python Logging Guide - Best Practices and Hands-on Examples
The default format for log records is SEVERITY: LOGGER: MESSAGE. ... argument exc_info to add exception information to log entries and ...
Read more >
amrayn/easyloggingpp - GitHub
It provides ability to write logs in your own customized format. It also provide support for logging your classes, third-party libraries, ...
Read more >
Java Logging Basics - The Ultimate Guide To Logging - Loggly
Java Logging Basics. This section presents Java logging basics, including how to create logs, popular logging frameworks, how to create some of the...
Read more >
Console log formatting - .NET - Microsoft Learn
In the preceding sample source code, the ConsoleFormatterNames.Simple formatter was registered. It provides logs with the ability to not only ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found