Integration with ScalaJSON

Finally after a long amount of time, ScalaJSON has been released on the scala-platform, follow on from SLIP-28 (its currently as a milestone release, see https://github.com/mdedetrich/scalajson). A proper release is expected to happen within a few days, as long as no major changes are expected. Its also a response to

You generally shouldn’t need or want to work with JSON ASTs directly. This is part of the reason I couldn’t personally care less about SLIP-28: I think JSON is a horrible serialization format, and I want to help people not have to think about JSON. If you absolutely have to work with JSON values directly, you at least shouldn’t have to worry about keeping track of traversal history or manually handling modification of deeply nested structures (i.e. you should be using a relatively nice API like ACursor or the facilities supported by Monocle in circe-optics).

What this means concretely: If something like SLIP-28 ever actually lands in the standard library, I’ll probably adopt it (assuming it doesn’t violate other design principles here—which is a huge assumption), but I hope most users won’t even notice that change.

In the design readme (https://github.com/circe/circe/blob/master/DESIGN.md)

This issue is to describe the likelyhood (and how) to integrate the AST into Circe (or if there is just going to be a circe-scalajson module which means no integration and just a conversion between the 2 modules). Ideally a proper integration would be great, however its understood if this is going to cause problems as the ScalaJSON AST is fairly different in design to the circe JSON AST (although most circe users shouldn’t be working directly on the Json AST). ScalaJSON AST is implemented as a proper Scala ADT where as the Json type in Circe has no ADT exposed

Here are the the different ways I envisage to integrate the ScalaJSON AST into Circe

• Just integrate scalajson.ast.unsafe.JValue in a low level way, and expose functions that allow you to retrieve this JValue (you can always get a scalajson.ast.JValue from a scalajson.unsafe.JValue using ScalaJSON)
• Integrate both scalajson.ast.unsafe.JValue and scalajson.ast.JValue into Circe, exposing scalajson.ast.JValue when it makes sense to do so. Note that doing this is a slight contradiction of Circe’s original design (i.e. users should not have to deal with the Json AST type directly) however it can provide extra benefit for users. Note that scalajson.ast.JValue doesn’t implement the whole possible JSON spectrum (see https://github.com/mdedetrich/scalajson#goals for more info)
• Don’t integrate at all, and instead have a module as part of Circe which converts from a Circe Json value to a ScalaJSON JValue and vice versa. Not ideal in terms of performance/speed since we are not using the AST directly (and also end users needing another module) however the easiest to work with from Circe’s POV.
• Least ideal situation, ScalaJSON provides a module to integrate with Circe which ScalaJSON will manage. This is least ideal because Circe contributors have a much better idea of how their release cycles work.

If there are any questions let me know. I am also happy to help with the integration, and there is room to do some extra changes providing they aren’t completely major (@travisbrown the change you requested in regards to number handling with overflow due to BigDecimal has already been implemented)

Also note there is an issue in jawn for integration the JSON into their parser, you can see it here https://github.com/non/jawn/issues/89

[travisbrown]

Thanks for opening this issue, @mdedetrich, and congrats on the release being almost ready!

Low-level adoption of the ScalaJSON AST

I still think it’s unlikely that circe will switch to using the ScalaJSON AST at a low level. I don’t intend to criticize the ScalaJSON design decisions, but they’re fairly different from circe’s, and I’m not convinced the performance benefits over a conversion module would be worth trying to make the two fit together. Specifically:

1. I’m still not sold on the safe / unsafe split. My goal for the AST in circe is to have something that can usefully represent any valid JSON value, supports a few popular distinctions that the JSON spec doesn’t make (negative zero, order preservation for object fields), and will never, ever throw exceptions, while still performing sufficiently well for most reasonable use cases. I think the current AST is more or less successful at doing this (and is still improving), and I haven’t seen any concrete evidence that the performance gains of an unsafe representation would offset the extra complexity.
2. I just don’t think having exceptions thrown on simple cases like JNumber("1.0").to[Int] or JNumber("1e2147483648").to[BigDecimal] (the “safe” versions of both of these throw) is a good user experience—I don’t really want to expose types that do this in the core circe API, and internally I’d have to do a lot of wrapping just to maintain the safe behavior I already have with io.circe.Json.

I’d consider revisiting this decision if there turns out to be overwhelming demand for optimally fast ScalaJSON integration, or if someone produces evidence that the performance supported by the unsafe representation is just vastly better than io.circe.Json for use cases people care about.

Conversions

I’m happy to add a ScalaJSON conversion module in circe right now, and I think we’d consider including ScalaJSON conversions in circe-core if there proves to be a lot of demand from circe adopters over the next few months.

Future integration

I’m also working on a major change to circe that would make this whole question a little less relevant. The key idea is that circe’s Decoder (and Encoder) will no longer use any particular AST directly: you’ll write your Decoder in an AST-agnostic way (using an API that’s like 95% source compatible with the current circe Decoder) and then use an AST-specific interpreter to decode your actual JSON representation (or your BSON or YAML representation, etc.).

This change (which I now think has maybe an 80% chance of actually happening) goes a step further in emphasizing encoders and decoders as the key components in circe, and de-emphasizing the AST and cursors. Once it’s done we can provide circe encoder and decoder interpreters for ScalaJSON and users who want to work with the ScalaJSON AST will never have to instantiate io.circe.Json values at all.

[mdedetrich]

I’m still not sold on the safe / unsafe split. My goal for the AST in circe is to have something that can usefully represent any valid JSON value, supports a few popular distinctions that the JSON spec doesn’t make (negative zero, order preservation for object fields), and will never, ever throw exceptions, while still performing sufficiently well for most reasonable use cases. I think the current AST is more or less successful at doing this (and is still improving), and I haven’t seen any concrete evidence that the performance gains of an unsafe representation would offset the extra complexity.

The safe/unsafe split is due to the unfortunate redefining of JSON by mr Crockford in his specification. The differences are largely due to ordering/duplicate keys in JObject (in Javascript which is where JSON came from, you can’t have duplicate keys/ordering in a Map, this was just added by Crockford when making the spec with weasel words such as “should” for justifying them). The reason behind the split is that there is a “sane” version of JSON which is what everyone should be using (this is the ScalaJSON standard AST), if they don’t use this “sane” version then they are probably going to break something, somewhere for someone. In reality there are reasons for these corner cases (i.e. ordering keys when it comes to serialization), hence the reason for unsafe.JValue.

Completely understand that the split AST is not someone that everyone is happy with, but having a combined correct AST comes at the cost of simplicity of implementation, which is a core premise of ScalaJSON AST

I just don’t think having exceptions thrown on simple cases like JNumber(“1.0”).to[Int] or JNumber(“1e2147483648”).to[BigDecimal] (the “safe” versions of both of these throw) is a good user experience—I don’t really want to expose types that do this in the core circe API, and internally I’d have to do a lot of wrapping just to maintain the safe behavior I already have with io.circe.Json.

I am actually not a fan on these and want to remove them, comment on https://github.com/mdedetrich/scalajson/issues/17 if you agree

I’m happy to add a ScalaJSON conversion module in circe right now, and I think we’d consider including ScalaJSON conversions in circe-core if there proves to be a lot of demand from circe adopters over the next few months.

Awesome thanks, I was expecting this response anyways 😉

I’m also working on a major change to circe that would make this whole question a little less relevant. The key idea is that circe’s Decoder (and Encoder) will no longer use any particular AST directly: you’ll write your Decoder in an AST-agnostic way (using an API that’s like 95% source compatible with the current circe Decoder) and then use an AST-specific interpreter to decode your actual JSON representation (or your BSON or YAML representation, etc.).

I actually like this change and I think the decoupling makes sense