Stuck on an issue?

These two libraries have had a good run over the last 3 years, but many of their original motivations for existing have since disappeared, and other parts of their design haven’t work out over time.

Automatic, deep compile-time derivation

The current implementation, in the derive/ subproject, is probably a dead end in terms of engineering: it is a rats nest of complexity that works ok for what it does now, but nobody understands how it works, and it is almost impossible to fix the existing bugs without causing regressions (i.e. the code is a total disaster).

This results in bugs like https://github.com/lihaoyi/Ammonite/issues/575, https://github.com/lihaoyi/upickle-pprint/issues/196, https://github.com/lihaoyi/upickle-pprint/issues/182, and may others that are beyond my ability to fix.

For JSON serialization, a partial workaround is to declare implicit read-writers in every case class’s companion objects via implicit val rw = upickle.default.macroRW[MyClass]. This helps avoid the nasty/buggy deep derivation logic entirely, and is something I have adopted in all my own projects. We could make this easier with a macro annotation, but fundamentally if we’re giving up on automatic deep compile-time derivation, we could just use something like https://github.com/playframework/play-json instead.

If we want to try a more principled approach to compile-time derivation, we should probably use Shapeless instead of doing it ourselves. In that case, we could also just use https://github.com/circe/circe which seems to work out of the box., or https://github.com/julienrf/play-json-derived-codecs

For pretty-printing, my main use case is for Ammonite on the JVM. In that case, using runtime/Java reflection is probably the right thing to do.

Scala-js Compatibility

uPickle was the first, and at the time only, cross-JS/JVM serialization library. Since then there have been countless others, including new ones like https://github.com/circe/circe and existing ones like https://github.com/playframework/play-json which have picked up cross-platform support.

Those other libraries are better maintained and more featureful, so since uPickle is no longer the only one with Scala.js support, we should probably focus our efforts/maintenance/contributions on one of the others.

I don’t really use PPrint much on Scala.js, so a Java-reflection-based PPrint that doesn’t support Scala.js isn’t an issue for me.

Simple API

Part of uPickle’s benefits is the simple API for serializing things:

import upickle.default._

write(1)                          ==> "1"

write(Seq(1, 2, 3))               ==> "[1,2,3]"

read[Seq[Int]]("[1, 2, 3]")       ==> List(1, 2, 3)

write((1, "omg", true))           ==> """[1,"omg",true]"""

And for looking things up:

json(8)("real").value == -9876.54321

Play-Json more or less already does the same thing:

@ Json.stringify(Json.toJson(1))
res4: String = "1"
@ Json.stringify(Json.toJson(Seq(1, 2, 3)))
res5: String = "[1,2,3]"
@ Json.fromJson[Seq[Int]](Json.parse("[1,2,3]"))
res6: play.api.libs.json.JsResult[Seq[Int]] = JsSuccess(List(1, 2, 3), )

While the API is more verbose, it should be trivial to add a helper function to simplify it. Other things, like tuples not being supported:

@ Json.stringify(Json.toJson((1, "omg", true)))
cmd7.sc:1: No Json serializer found for type (Int, String, Boolean). Try to implement an implicit Writes or Format for this type.
val res7 = Json.stringify(Json.toJson((1, "omg", true)))
                                     ^
Compilation Failed

Should also be easy to add.

Similarly, play-json has a somewhat inconsistent .apply/\ based API, but that can also be fixed relatively easily to provide the same lookup-syntax that uPickle currently has:

• https://github.com/playframework/play-json/pull/66

Overall, play-json is probably similar enough to uPickle at this point, with the new Scala.js support, that it might be easier to start from it as a base and port the simple API on top of it, rather than trying to wrangle uPickle’s internals into a simpler state. I’m not familiar enough with Circe to say whether or not it’s the same, but likely they solved many of the same problems, and it may also be easier to prettify circe rather than cleaning up uPickle’s internals

Additional Features

It would be nice to provide things like:

• Streaming reading/writing of JSON from e.g. java.io.{Input,Output}Streams
• “deep” xpath-style lookup, a better syntax for creating JSON fragments
• Better error messages for parsing and validation

These are all things that Circe/Play-Json provide to varying degrees, and it would be nice if we could leverage that rather than re-implementing all the same stuff in uPickle

Overall, uPickle and PPrint have had a good run over the last 3 years, but I think the design of these two libraries is probably a dead end. The best way forward is probably to port all the syntactic niceties over to play-json, whether upstream or as a fork, and implementing a Java-reflection-based pretty-printer for use in Ammonite.

Things that would make play-json work as a mostly-drop-in replacement for uPickle:

• Allow for the same field-lookup syntax playframework/play-json#66
• Make JsArray take an IndexedSeq to enforce efficient indexed lookups https://github.com/playframework/play-json/pull/67
• Make JsNumber take a Double or a String instead of a BigDecimal, to avoid the huge memory overhead of the BigDecimal data structure
• Document how automatic, shallow sealed-trait derivation works, or how to manually defined shallow sealed-trait serializers as in http://www.lihaoyi.com/upickle-pprint/upickle/#ManualSealedTraitPicklers
• Document how to configure serialization, as uPickle allows in http://www.lihaoyi.com/upickle-pprint/upickle/#CustomConfiguration
• ??? Remove dependency on Jackson/Java/Reflection in favor of something like https://github.com/non/jawn, which will allow the same streaming/async parsing Jackson gives you but work on both JVM/JS

There may be a few others, but it’s a relatively short list that would make play-json work for 99% of uPickle’s use cases