How to implement a PatternMaskingLayout for a LoggingEventCompositeJsonEncoder

Hi. I have the following problem. I’m using the LoggingEventCompositeJsonEncoder to generate JSON logs, but I found out that we have lots of confidential information that we want to filter out (basically just put *** in the corresponding key).

https://www.schibsted.pl/blog/logback-pattern-gdpr/

One idea here is to use this:

      <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
           <layout class="com.schibsted.payment.logback.PatternMaskingLayout">

However, this will not log everything nicely in JSON.

Is there a way to accomplish what I want?

[rusucosmin]

I’ve written a simple version which worked for my specific needs: https://github.com/logstash/logstash-logback-encoder/pull/375.

However, I don’t feel like it’s yet great. I might need some feedback from @phasebash.

The problem is that if I try to log some specific objects (for example, I wanted to log JSONObject s and JSONArrays. I ended up doing something like this in the writeObject method:

    } else if (JSONObject.class.isAssignableFrom(cls)) { // sub-class
      return sanitize((JSONObject) ob);
    } else if (JSONArray.class.isAssignableFrom(cls)) { // sub-class
      return sanitize((JSONArray) ob);

This is clearly not a generic use and you’ll have to implement overloaded sanitize methods. I’m wondering what would be an alternative to this?

[philsttr]

Yes definitely!

And I’d love if the logic to detect what should be masked would be customizable. And logstash-logback-encoder provided some detectors for standard use cases, I can think of three cases:

1. by field name, where if a field name in any object at any path matches a list of global field names to mask, then mask it
2. by sub path, where if the path of a field matches a list subpaths to mask, then mask it
3. by field value, where if a value matches a regex of values to mask

1 and 2 could potentially be combined if it’s coded such that a single field name is just a subpath with one element.

Probably need to handle masking any type of field value (e.g. number fields, string fields, array fields, and object fields.)