Tuesday, September 11, 2012

Full (de)serialization for Play Json using general purpose macros

I've released a new version of akmacros library for Scala 2.10. The release includes a new macro called 'factory'. Thanks to the new macro it is possible to construct a class with a public default constructor just passing a function to the generated factory of the class. The function receives symbol of an argument of the constructor and returns value for this argument. If function returns None, then default value for the argument is evaluated (provided that it is defined for the given argument). Here is the example:
    scala> import info.akshaal.clazz._
    import info.akshaal.clazz._

    scala> case class Record (name: String, twitter: Option[String] = None)
    defined class Record

    scala> val recordFactory = factory[Any, Record]('castValue)
    recordFactory: (Symbol => Option[Any]) => Record = <function1>

    scala> val args = Map('name -> "Evgeny")
    args: scala.collection.immutable.Map[Symbol,String] = Map('name -> Evgeny)

    scala> recordFactory(args.get)
    res0: Record = Record(Evgeny,None)

Writing less boilerplate code for Play Json

Just to demonstrate the way macro functions can simplify your code I've created a small project on github. The project includes Json library from the Play2.0 framework mostly as-is. The only changes I did are related to making it work on scala 2.10 (upgraded it to patched jerkson and so on). You might be interested in this file. It is the only file that I implemented in that project in order to add support for macros to the Play Json. Following the same pattern, you can easily use almost any other Json library like this.

Lets look what it gives you without any reflections.. Lets start with something simple:
    scala> import info.akshaal.json.play._
    import info.akshaal.json.play._

    scala> import play.api.libs.json._
    import play.api.libs.json._

    scala> case class Simple(str: String, num: Int = new Random().nextInt())
    defined class Simple

    scala> implicit val simpleJsFactory = factory[Simple]('fromJson)
    simpleJsFactory: (Symbol => Option[play.api.libs.json.JsValue]) => Simple = <function1>

    scala> implicit val simpleJsFields = allFields[Simple]('jsonate)
    simpleJsFields: List[info.akshaal.clazz.Field[Simple,play.api.libs.json.JsValue,None.type]] = List(Field(str,<function1>,None), Field(num,<function1>,None))
We can serialize and then deserialize:
    scala> val obj = new Simple("123", 5)
    obj: Simple = Simple(123,5)

    scala> val objJs = Json.toJson(obj)
    objJs: play.api.libs.json.JsValue = {"str":"123","num":5}

    scala> val obj2 = Json.fromJson[Simple](objJs)
    obj2: Simple = Simple(123,5)
That was easy. Now lets try to leverage default value feature of the case class:
    scala> val halfObjJs = Json.parse(""" {"str":"test"} """)
    halfObjJs: play.api.libs.json.JsValue = {"str":"test"}

    scala> val halfObj = Json.fromJson[Simple](halfObjJs)
    halfObj: Simple = Simple(test,-813663852)

    scala> val halfObj = Json.fromJson[Simple](halfObjJs)
    halfObj: Simple = Simple(test,-948806581)

    scala> val halfObj = Json.fromJson[Simple](halfObjJs)
    halfObj: Simple = Simple(test,428745442)
Note that we parsed halfObjJs three times. And 'num' was always different. That is because value for 'num' is missing in the halfObjJs and so the expression for default value was used (which is Random().nextInt, see defintion of Simple class). How about parametrized classes? Lets try:
    scala> case class Event[T](kind: String, payloads: T)
    defined class Event

    scala> implicit def eventJsFields[T: Writes] = allFields[Event[T]]('jsonate)
    eventJsFields: [T](implicit evidence$1: play.api.libs.json.Writes[T])List[info.akshaal.clazz.Field[Event[T],play.api.libs.json.JsValue,None.type]]

    scala> implicit def eventJsFactory[T: Reads] = factory[Event[T]]('fromJson)
    eventJsFactory: [T](implicit evidence$1: play.api.libs.json.Reads[T])(Symbol => Option[play.api.libs.json.JsValue]) => Event[T]
And test:
    scala> val event = Event(kind = "strange", payloads = obj)
    event: Event[Simple] = Event(strange,Simple(123,5))

    scala> val eventJs = Json.toJson(event)
    eventJs: play.api.libs.json.JsValue = {"kind":"strange","payloads":{"str":"123","num":5}}

    scala> val event3 = Json.fromJson[Event[Simple]](eventJs)
    event3: Event[Simple] = Event(strange,Simple(123,5))
In the real world it is unlikely that you will use completely unrestricted types (like T in the example above) in your case classes with json. It is more likely that there will be a (sealed) trait and some set of its subtypes. Lets see how it works. First, we will define some more case classes:
    sealed trait Message
    case class QuitMessage(msg: String) extends Message
    case class Heartbeat(id: Int) extends Message

    case class Messages(userId: Int, messages: List[Message] = List.empty)
Now, lets define how to serialize our new case classes:
    implicit def messageWrites = matchingWrites[Message] {
        case m: QuitMessage => quitMessageJsFields.toWrites.writes(m)
        case m: Heartbeat   => heartbeatJsFields.toWrites.writes(m)

    implicit def quitMessageJsFields = allFields[QuitMessage]('jsonate)
    implicit def heartbeatJsFields = allFields[Heartbeat]('jsonate)
    implicit def messagesJsFields = allFields[Messages]('jsonate)
And if deserialization is needed, lets define how to do it:
    implicit def quitMessageJsFactory = factory[QuitMessage]('fromJson)
    implicit def heartbeatJsFactory = factory[Heartbeat]('fromJson)
    implicit def messagesJsFactory = factory[Messages]('fromJson)

    implicit def messageReads: Reads[Message] =
            jsHas('msg) -> quitMessageJsFactory,
            jsHas('id) -> heartbeatJsFactory
Finally, lets try it:
    scala> val event5 =
              Event(kind = "t1",
                    payloads = Messages(userId = 4, messages = List(Heartbeat(5), QuitMessage("bye!"))))
    event5: Event[Messages] = Event(t1,Messages(4,List(Heartbeat(5), QuitMessage(bye!))))

    scala> val event5Js = Json.toJson(event5)
    event5Js: play.api.libs.json.JsValue = {"kind":"t1","payloads":{"userId":4,"messages":[{"id":5},{"msg":"bye!"}]}}
Pay attention, that Json for QuitMessage and Heartbeat classes has no type information or anything! It's just plain json with domain fields only. That's why messageReads has those jsHas occurances in its implementation, that is a little help for identifying which json object is what subtype of Message. Lets see that reading from json still works:
    scala> val event6 = Json.fromJson[Event[Messages]](event5Js)
    even6: Event[Messages] = Event(t1,Messages(4,List(Heartbeat(5), QuitMessage(bye!))))
Actually there is another way to do the same. You can inject an extra field into json object when serializing one of subtypes of an abstract type and use it as a guidance for reconstructing objects from json. It is really easy. Lets modify messageReads and messageWrites like this:
    implicit def messageWrites = matchingWrites[Message] {
        case m: QuitMessage => quitMessageJsFields.extra('type -> 'quit).toWrites.writes(m)
        case m: Heartbeat   => heartbeatJsFields.extra('type -> 'heart).toWrites.writes(m)

    implicit def messageReads: Reads[Message] =
            jsHas('type -> 'quit)  -> quitMessageJsFactory,
            jsHas('type -> 'heart) -> heartbeatJsFactory
Now we test the new implementation.
    scala> val messages = List(Heartbeat(1), QuitMessage("Hello"), Heartbeat(99))
    messages: List[Product with Serializable with Message] = List(Heartbeat(1), QuitMessage(Hello), Heartbeat(99))

    scala> val messagesJs = Json.toJson(messages)
    messagesJs: play.api.libs.json.JsValue = [{"type":"heart","id":1},{"type":"quit","msg":"Hello"},{"type":"heart","id":99}]

    scala> val messages2 = Json.fromJson[List[Message]](messagesJs)
    messages2: List[Message] = List(Heartbeat(1), QuitMessage(Hello), Heartbeat(99))
That's not all. What if there is an information in a case class you don't want to reveal in JSON? Consider the following case class. I will annotate fields that are safe to export by Ok annotation:
    class Ok extends annotation.StaticAnnotation

    case class User(@Ok login: String,
                    @Ok fullName: String,
                    @Ok messages: Int = 0,
                    passwordHash: Option[String] = None)
Now it's quite natural to define json fields like this:
    implicit val userJsFields = annotatedFields[User, Ok]('jsonate)
Let see it in action:
    |       val user = User(login = "akshaal",
    |                       fullName = "Evgeny Chukreev",
    |                       messages = 10,
    |                       passwordHash = Some("4d18758602c08243d7c08f8c9e4463b0"))
    user: User = User(akshaal,Evgeny Chukreev,10,Some(4d18758602c08243d7c08f8c9e4463b0))

    scala> val userJs = Json.toJson(user)
    userJs: play.api.libs.json.JsValue = {"login":"akshaal","fullName":"Evgeny Chukreev","messages":10}
It works.. But you can do more. Recall (if you looked at the implementation) that jsonate function was defined like this:
    def jsonate[T: Writes](t: T, args: Any): JsValue = Json.toJson(t)
The function is used to make a json value out of field's value. It is applied to each field. So you can define your our function that does post-processing (pre-processing?) and use it with a macro. The second argument (args) might be a parameter set given to annotation, this gives you even more power for writing complex json serialization easily.

And not only JSON. Using the same approach you can (de)serialize object from/to XML...

Pros of akmacros-json

  • Domain classes are separated from any notion of JSON
  • Full control over serialization/deserialization
  • Easy to use
  • Easy to extend or implement your own
  • No runtime reflections used

Cons of akmacros-json

  • Depends on Jerkson which is officially unavailable for Scala 2.10 (you need to build it from my fork (in order to build it you will need also this forked project))
  • Includes a copy of Play Json
  • Scala 2.10-M7 has many bugs related to implicits, value classes... so implementation as you might have noticed is not perfect in terms of performance (defs are used instead of vals)
I hope things will change really soon with the release of Scala 2.10.

About general purpose macros

This tiny macros addition is built on top of Play JSON (which is built on top of Jerkson (which is built on top of Jackson)) and akmacros (which doesn't have dependencies). Checkout 78 lines long implementation here. See https://github.com/akshaal/akmacros for a bit more information about using akmacros with sbt.

No comments:

Post a Comment