Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Default values not working with @key annotation in Scala 3

See original GitHub issue

Issue Description

case class Person(
                     @key("first_name") firstName: String = "N/A",
                     @key("last_name") lastName: String)
object Person {
    implicit val rw: RW[Person] = macroRW

val json = """{"last_name": "Snow"}"""
val p = read[Person](json)

This code fragment executes with exception:

Exception in thread "main" java.lang.ExceptionInInitializerError
	at com.nitka.rb.RBSyncApp.main(RBSyncApp.scala)
Caused by: upickle.core.AbortException: missing keys in dictionary: first_name at index 20
	at ujson.CharParser$$anon$1.applyOrElse(CharParser.scala:343)
	at ujson.CharParser$$anon$1.applyOrElse(CharParser.scala:341)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
	at ujson.CharParser.liftedTree1$1(CharParser.scala:496)
	at ujson.CharParser.tryCloseCollection(CharParser.scala:496)
	at ujson.CharParser.parseNested(CharParser.scala:462)
	at ujson.CharParser.parseTopLevel0(CharParser.scala:323)
	at ujson.CharParser.parseTopLevel(CharParser.scala:307)
	at ujson.CharParser.parse(CharParser.scala:59)
	at ujson.StringParser$.transform(StringParser.scala:28)
	at ujson.StringParser$.transform(StringParser.scala:28)
	at ujson.Readable$fromTransformer.transform(Readable.scala:13)
	at upickle.core.TraceVisitor$.withTrace(TraceVisitor.scala:18)
	at upickle.default$.read(Api.scala:133)
	at com.nitka.rb.RBSyncApp$.<clinit>(RBSyncApp.scala:76)
	... 1 more
Caused by: upickle.core.Abort: missing keys in dictionary: first_name
	at com.nitka.rb.RBSyncApp$$anon$1.make(RBSyncApp.scala:72)
	at com.nitka.rb.RBSyncApp$$anon$1.make(RBSyncApp.scala:72)
	at upickle.implicits.CaseClassReaderPiece$$anon$1.visitEnd(CaseClassReader.scala:30)
	... 16 more

Reproduced with uPickle version 1.4.0

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:9 (6 by maintainers)

github_iconTop GitHub Comments

htmldougcommented, Sep 9, 2021

Thanks the response and for understanding so well.

@russellremple took the upickle scala 3 macros and tweaked them a bit in, so they should be quite similar. It’s exciting to hear that they work for scala 3 enums, particularly since we didn’t even test for that yet! I wonder what the difference is between the implementations.

Many of our macro changes were aimed at runtime performance. We found that the scala 3 upickle macro implementations have something like 2% of the throughput of the scala 2 macros. There’s some low hanging fruit (especially around default values), but IIRC, we hit a frustrating ceiling at ~50%-75% vs scala 2. inline is delightfully simple, but I suspect we’ll need the lower level reflection APIs to reach performance parity. I’m considering the weepickle implementations temporary until I get around to exploring the new macro system deeper. We both have some reading to do. 😃

porting bug-fixes and improvements back would be really great for upickle.

For sure. It’s great for everyone. Besides minor behavior differences and the weepickle naming (Writer == From, Reader == To, ReadWriter == FromTo), all the bones are the same. I want to keep cross-porting as straightforward as it can be.

htmldougcommented, Sep 5, 2021

mill-mima is great to see! That was the biggest missing piece that finally drove weePickle from mill => sbt.

Do you think we can unify weepickle and upickle again?

Simple question, complicated answer. TL;DR: I’m not sure that unifying would be worth it for either project.

upickle is already getting the bug fixes from weepickle which are easily portable as well as notable features like #286. upickle also has access to the weepickle+jackson ecosystem of YAML, CBOR, Smile, XML (such that it is), etc. via the bridges in, if desired.

I understand there’s a lot going on with the scala 2 macros. Regarding the fix for this issue, the weepickle’s macros have diverged a bit. I’m not sure you would want all our changes to the scala 2 macros as they are. For example, the section to support case classes with > 64 fields and maintain binary backwards compatibility is particularly ugly. In the past, the upickle approach has been to make the breaking changes, but keep the implementation simple. So for this issue, I’d just port the part you care about, or just ignore weepickle entirely–whichever is easier.

Satisfying rallyhealth/weepickle’s needs would impose some additional constraints on upickle. For context, Rally has hundreds of internal scala repos using weepickle with an inter-dependency tree and different paces of development and staffing. Taking breaking changes with micro-repos is much more expensive than the databricks monorepo setup, since different modules may be compiled against different (binary incompatible) versions of the same library, resulting in runtime exceptions.

So, we shade (at publish time) all of our internal libraries that are meant to be used by other libraries and prefer third party libraries that do the same. MiMa + semver is not enough to avoid dependency hell, but shading is.

Is shading something that upickle would be willing to adopt on the next major release/mima failure? It’s a big ask. Among other things, it might require a major version bump of Haoyi’s book to update the examples.

weepickle has also made other minor tradeoffs in complexity/performance and behavior idiosyncrasies/ease of migration from play-json that we might be able work through to unify, but shading is the first major hurdle.

A second factor is that Rally has invested a fair amount in migrating from play-json to weepickle because: 1. it stopped wasting developer time on dependency hell, 2. stopped the OOMEs in prod apps, 3. improved response times. It’d be hard to make a business case to migrate 338 repos to another json library so soon, particularly without equally compelling benefits over weepickle.

I’m a huge fan of the com-lihaoyi ecosystem. It’s a great fit for monorepos or leaf nodes of the repo dependency tree. It’s just not a clear fit for our non-leaf libraries nodes due to our potential downstream costs of breaking changes in v1 (yay mill-mima!) or the threat of an unshaded v2.

I know this was a long answer. I’d love to hear your thoughts.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Default Parameter Values | Tour of Scala
Scala provides the ability to give parameters default values that can be used to allow a caller to omit those parameters. The parameter...
Read more >
How to set default values for Scala method parameters
Problem. You want to set default values for Scala method parameters so ... Specify the default value for parameters in the method signature....
Read more >
Macro annotation with default arguments - Stack Overflow
The logic in ??? seems to become very nasty: I need to handle both positional and named parameters, no simple access to the...
Read more >
µPickle 2.0.0
uPickle does not support Scala 2.10; only 2.11/2.12/2.13/3.x are supported. ... If a field at serialization time has the same value as the...
Read more >
Annotations in Java - GeeksforGeeks
Annotations do not change the action of a compiled program. ... The 3 values that the @Retention annotation can have:.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found