Problem deserializing Java 8 `Optional` with Polymorphic Deserialization with Jackson-databind 2.12 vs 2.8

Describe the bug We recently tried to upgrade our application’s Jackson dependencies from 2.8.x to 2.12.x. After the upgrade though, we noticed some incompatible behaviors regarding the serialization and de-serialization results.

Code Setup

We have a POJO called RequestParameter, and it contains a metadata field:

private final Optional<Map<String, Object>> metadata;

This field is initialized in the constructor like the following:

this.metadata = Optional.of(ImmutableMap.copyOf(metadata.get()));

(The ImmutableMap here is the Guava ImmutableMap).

We use Mixins with the following configurations to specify the default type resolving behavior for the Map interface: (This might not be the best way to specify the default resolving behaviors for polymorphic fields, but our code was already setup this way…)

    @JsonTypeInfo(use = JsonTypeInfo.Id.NONE, defaultImpl = HashMap.class)
    public static class MapMixin {}

Note that we are using JsonTypeInfo.Id.NONE here to drop the type information for Map.

How we create our ObjectMapper:

        ObjectMapper mapper = new ObjectMapper()
                  .setSerializationInclusion(JsonInclude.Include.NON_NULL)
                  .enableDefaultTyping(DefaultTyping.NON_FINAL, As.PROPERTY)
                  .disable(SerializationFeature.FAIL_ON_EMPTY_BEANS)
                  .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);

        mapper.addMixInAnnotations(ServerMessage.class, FrameworkMessageMixin.class);
        mapper.addMixInAnnotations(ClientMessage.class, FrameworkMessageMixin.class);
        mapper.registerSubtypes(...);

        // some more configurations
        objectMapper.enableDefaultTypingAsProperty(DefaultTyping.NON_CONCRETE_AND_ARRAYS,
                JsonCoderSpecification.TYPE_PROPERTY);
        objectMapper.setVisibility(PropertyAccessor.FIELD, JsonAutoDetect.Visibility.ANY);
        objectMapper.setVisibility(PropertyAccessor.GETTER, JsonAutoDetect.Visibility.NONE);
        objectMapper.setVisibility(PropertyAccessor.IS_GETTER, JsonAutoDetect.Visibility.NONE);
        objectMapper.registerModule(new Jdk8Module());
        objectMapper.addMixIn(Map.class, MapMixin.class);
        objectMapper.addMixIn(List.class, ListMixin.class);
        objectMapper.addMixIn(Set.class, SetMixin.class);

Observed Behaviors

Let’s say we have a RequestParameter POJO, and it’s metadata field has one entry in it: "testKey":"testValue". Below are the serialization and deserialization results for this POJO under different Jackson versions:

With 2.8.x

// Serialized result
{"__type__":"RequestParameter", "metadata":["java.util.Optional",{"testKey":"testValue"}]}

// Deserialized result for the metadata field (based on POJO toString)
metadata=Optional[{testKey=testValue}]

With 2.12.x (and even 2.10.x)

// Serialized result
{"__type__":"RequestParameter", "metadata":{"__type__":"com.google.common.collect.SingletonImmutableBiMap","testKey":"testValue"}}

// Deserialized result for the metadata field (based on POJO toString)
metadata=Optional[{__type__=com.google.common.collect.SingletonImmutableBiMap, testKey=testValue}]

As we can see, with 2.12.x, the JsonTypeInfo.Id.NONE specified in the Mixin is not really working, as the class/type info is still somehow added during the serialization process. The Optional wrapper for the metadata field is also somehow missing in the serialized result. As a result, when we try to deserialize the String back to the RequestParameter POJO, we end up having a Map with 2 entries, which is inaccurate.

Please help look into this incompatible behavior. Thank you!

Version information Old: Jackson-databind = 2.8.x; Jackson-datatype-jdk8 = 2.8.x;

New: Jackson-databind = 2.12.x; Jackson-datatype-jdk8 = 2.12.x;

**Issues that are related

• https://github.com/FasterXML/jackson-modules-java8/issues/59
• https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.10#changes-compatibility

[cowtowncoder]

I’ll make this quick: Polymorphic Type Information for referential types like Optional should always be for content value, and never for “Optional” itself. This is the behavior expected with 2.12 and later, regardless of what 2.8 did. Unlike with other Container types (Collections, Maps), there is no marker (Array, Object) on which Type Id could be attached, so the choice is between contents (almost always more important) or nominal Optional wrapper (rarely more important one).

So, the expectation is that Type Id for content value, if any, should be output (assuming Polymorphic Typing is enabled for (nominal) value type – if it’s not, no Type Id should be written or expected).

Put another way, Type Id for Optional should quite literally never be written by serialization.

[coriedai]

Updated POJO to remove Lombok dependencies:

import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.annotation.JsonPOJOBuilder;

@JsonIgnoreProperties(ignoreUnknown = true)
@JsonDeserialize(builder = RequestParameter.RequestParameterBuilder.class)
public class RequestParameter {

  private final AtomicReference<Map<String, Object>> metadata;

  private RequestParameter(final AtomicReference<Map<String, Object>> metadata) {
      this.metadata = metadata;
  }

  public boolean equals(Object o) {
    // self check
    if (this == o)
      return true;
    // null check
    if (o == null)
      return false;
    // type check and cast
    if (getClass() != o.getClass())
      return false;

    RequestParameter parameter = (RequestParameter) o;

    return this.metadata.get().equals(parameter.metadata.get());
  }

  public String toString() {
    return "RequestParameter=(metadata=" + metadata + ")";
  }

  public static RequestParameterBuilder builder() {
    return new RequestParameterBuilder();
  }

  @JsonPOJOBuilder(withPrefix = "")
  public static class RequestParameterBuilder {
    private AtomicReference<Map<String, Object>> metadata;

    RequestParameterBuilder() {}

    public RequestParameterBuilder metadata(AtomicReference<Map<String, Object>> metadata) {
      this.metadata = metadata;
      return this;
    }

    public RequestParameter build() {
      return new RequestParameter(this.metadata);
    }
  }
}

The test results I posted earlier can be reproduced with both jackson 2.8 and 2.12.

As a side note, based on the current test POJO and test suite, if I change the AtomicReference to Optional, and add back the Jdk8Module dependency, then the original inconsistent results can be reproduced.

With 2.8.x, the test would pass:

    [junit] serializedResult: {"__type__":"RequestParameter","metadata":["java.util.Optional",{"testKey":"testValue"}]}
    [junit] deserializedResut: RequestParameter=(metadata=Optional[{testKey=testValue}])

With 2.12.x, the test would fail:

    [junit] serializedResult: {"__type__":"RequestParameter","metadata":{"__type__":"java.util.HashMap","testKey":"testValue"}}
    [junit] deserializedResut: RequestParameter=(metadata=Optional[{__type__=java.util.HashMap, testKey=testValue}])

    [junit] expected:<RequestParameter=(metadata=Optional[{testKey=testValue}])> but was:<RequestParameter=(metadata=Optional[{__type__=java.util.HashMap, testKey=testValue}])>
    [junit] junit.framework.AssertionFailedError: expected:<RequestParameter=(metadata=Optional[{testKey=testValue}])> but was:<RequestParameter=(metadata=Optional[{__type__=java.util.HashMap, testKey=testValue}])>