Default values not working with @key annotation in Scala 3

case class Person(
                     @key("first_name") firstName: String = "N/A",
                     @key("last_name") lastName: String)
object Person {
    implicit val rw: RW[Person] = macroRW
}

val json = """{"last_name": "Snow"}"""
val p = read[Person](json)

This code fragment executes with exception:

Exception in thread "main" java.lang.ExceptionInInitializerError
	at com.nitka.rb.RBSyncApp.main(RBSyncApp.scala)
Caused by: upickle.core.AbortException: missing keys in dictionary: first_name at index 20
	at ujson.CharParser$$anon$1.applyOrElse(CharParser.scala:343)
	at ujson.CharParser$$anon$1.applyOrElse(CharParser.scala:341)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
	at ujson.CharParser.liftedTree1$1(CharParser.scala:496)
	at ujson.CharParser.tryCloseCollection(CharParser.scala:496)
	at ujson.CharParser.parseNested(CharParser.scala:462)
	at ujson.CharParser.parseTopLevel0(CharParser.scala:323)
	at ujson.CharParser.parseTopLevel(CharParser.scala:307)
	at ujson.CharParser.parse(CharParser.scala:59)
	at ujson.StringParser$.transform(StringParser.scala:28)
	at ujson.StringParser$.transform(StringParser.scala:28)
	at ujson.Readable$fromTransformer.transform(Readable.scala:13)
	at upickle.Api.read$$anonfun$1(Api.scala:37)
	at upickle.core.TraceVisitor$.withTrace(TraceVisitor.scala:18)
	at upickle.Api.read(Api.scala:37)
	at upickle.Api.read$(Api.scala:17)
	at upickle.default$.read(Api.scala:133)
	at com.nitka.rb.RBSyncApp$.<clinit>(RBSyncApp.scala:76)
	... 1 more
Caused by: upickle.core.Abort: missing keys in dictionary: first_name
	at com.nitka.rb.RBSyncApp$$anon$1.make(RBSyncApp.scala:72)
	at com.nitka.rb.RBSyncApp$$anon$1.make(RBSyncApp.scala:72)
	at upickle.implicits.CaseClassReaderPiece$$anon$1.visitEnd(CaseClassReader.scala:30)
	... 16 more

Reproduced with uPickle version 1.4.0

[htmldoug]

Thanks the response and for understanding so well.

@russellremple took the upickle scala 3 macros and tweaked them a bit in https://github.com/rallyhealth/weePickle/pull/81, so they should be quite similar. It’s exciting to hear that they work for scala 3 enums, particularly since we didn’t even test for that yet! I wonder what the difference is between the implementations.

Many of our macro changes were aimed at runtime performance. We found that the scala 3 upickle macro implementations have something like 2% of the throughput of the scala 2 macros. There’s some low hanging fruit (especially around default values), but IIRC, we hit a frustrating ceiling at ~50%-75% vs scala 2. inline is delightfully simple, but I suspect we’ll need the lower level reflection APIs to reach performance parity. I’m considering the weepickle implementations temporary until I get around to exploring the new macro system deeper. We both have some reading to do. 😃

porting bug-fixes and improvements back would be really great for upickle.

For sure. It’s great for everyone. Besides minor behavior differences and the weepickle naming (Writer == From, Reader == To, ReadWriter == FromTo), all the bones are the same. I want to keep cross-porting as straightforward as it can be.

[htmldoug]

mill-mima is great to see! That was the biggest missing piece that finally drove weePickle from mill => sbt.

Do you think we can unify weepickle and upickle again?

Simple question, complicated answer. TL;DR: I’m not sure that unifying would be worth it for either project.

upickle is already getting the bug fixes from weepickle which are easily portable as well as notable features like #286. upickle also has access to the weepickle+jackson ecosystem of YAML, CBOR, Smile, XML (such that it is), etc. via the bridges in https://github.com/rallyhealth/weePickle/pull/87, if desired.

I understand there’s a lot going on with the scala 2 macros. Regarding the fix for this issue, the weepickle’s macros have diverged a bit. I’m not sure you would want all our changes to the scala 2 macros as they are. For example, the section to support case classes with > 64 fields and maintain binary backwards compatibility is particularly ugly. In the past, the upickle approach has been to make the breaking changes, but keep the implementation simple. So for this issue, I’d just port the part you care about, or just ignore weepickle entirely–whichever is easier.

Satisfying rallyhealth/weepickle’s needs would impose some additional constraints on upickle. For context, Rally has hundreds of internal scala repos using weepickle with an inter-dependency tree and different paces of development and staffing. Taking breaking changes with micro-repos is much more expensive than the databricks monorepo setup, since different modules may be compiled against different (binary incompatible) versions of the same library, resulting in runtime exceptions.

So, we shade (at publish time) all of our internal libraries that are meant to be used by other libraries and prefer third party libraries that do the same. MiMa + semver is not enough to avoid dependency hell, but shading is.

Is shading something that upickle would be willing to adopt on the next major release/mima failure? It’s a big ask. Among other things, it might require a major version bump of Haoyi’s book to update the examples.

weepickle has also made other minor tradeoffs in complexity/performance and behavior idiosyncrasies/ease of migration from play-json that we might be able work through to unify, but shading is the first major hurdle.

A second factor is that Rally has invested a fair amount in migrating from play-json to weepickle because: 1. it stopped wasting developer time on dependency hell, 2. stopped the OOMEs in prod apps, 3. improved response times. It’d be hard to make a business case to migrate 338 repos to another json library so soon, particularly without equally compelling benefits over weepickle.

I’m a huge fan of the com-lihaoyi ecosystem. It’s a great fit for monorepos or leaf nodes of the repo dependency tree. It’s just not a clear fit for our non-leaf libraries nodes due to our potential downstream costs of breaking changes in v1 (yay mill-mima!) or the threat of an unshaded v2.

I know this was a long answer. I’d love to hear your thoughts.