Saturday, March 8, 2014

Setting up direct access to a block device from VM in Xen XCP (Debian 7.4 (wheezy))

There is no good documentation for XCP at the moment. Amount of documentation for such a complicated project is close to nothing when it comes to XCP on Debian (project Kronos). Citrix XenServer has a better support but I don't use XenServer, I use Debian. Therefore almost any simple task starts with "hours" of search in internet.
Normally you don't need direct access to a block device from a virtual machine, but in some cases it is very to convenient to use one. One of such cases is a file/backup server which needs access to a raid or just a huge hard disk with an existing file system.
Here is what should be done in order to make it work.
1. Create a directory which will contain links to the block devices visible in XEN.
mkdir /srv/xen-devices

2. Create storage repository for the block devices
xe sr-create name-label="Block devices" name-description="Block devices which we want to use directly in XEN" type=udev content-type=disk device-config:location=/srv/xen-devices
assign displayed uuid to shell variable SR

3. [Optional] If step 2 fails with the error message:
The SR could not be connected because the driver was not recognised.
driver: udev

then you need to fix it. Do the following:
cd /usr/lib/xcp/sm/
ln -s udevSR.py udevSR
/usr/lib/xcp/bin/xe-toolstack-restart
And then try again step 2.

4. Find a device you want to make visible in the Xen, for example, like this:
ls -l /dev/disk/by-path

Link it
ln -s /dev/disk/by-id/ata-some-device /srv/xen-devices/some-device

5. Rescan our storage repository
xe sr-scan uuid=$SR

6. List VDIs of the storage repository and spot the one with "Unrecognised bus type"
xe vdi-list sr-uuid=$SR

assign VDI¨s uuid to the VDI variable in shell.

7. Set name and label for the VDI to whatever you like:
xe vdi-param-set uuid=$VDI name-description="A real disk" 
xe vdi-param-set uuid=$VDI name-label="R-disk"

8. Create VBD which assign the VDI to the VM
xe vbd-create device=1 vm-uuid=$VM vdi-uuid=$VDI bootable=false mode=RW type=Disk
assign display uuid to the VBD variable

9. Attach it to the VM
xe vbd-plug uuid=$VBD

That's all. Now you can log into the virtual machine and check with 'dmesg' that a new device is available.


Tuesday, September 11, 2012

Full (de)serialization for Play Json using general purpose macros

I've released a new version of akmacros library for Scala 2.10. The release includes a new macro called 'factory'. Thanks to the new macro it is possible to construct a class with a public default constructor just passing a function to the generated factory of the class. The function receives symbol of an argument of the constructor and returns value for this argument. If function returns None, then default value for the argument is evaluated (provided that it is defined for the given argument). Here is the example:
    scala> import info.akshaal.clazz._
    import info.akshaal.clazz._

    scala> case class Record (name: String, twitter: Option[String] = None)
    defined class Record

    scala> val recordFactory = factory[Any, Record]('castValue)
    recordFactory: (Symbol => Option[Any]) => Record = <function1>

    scala> val args = Map('name -> "Evgeny")
    args: scala.collection.immutable.Map[Symbol,String] = Map('name -> Evgeny)

    scala> recordFactory(args.get)
    res0: Record = Record(Evgeny,None)

Writing less boilerplate code for Play Json

Just to demonstrate the way macro functions can simplify your code I've created a small project on github. The project includes Json library from the Play2.0 framework mostly as-is. The only changes I did are related to making it work on scala 2.10 (upgraded it to patched jerkson and so on). You might be interested in this file. It is the only file that I implemented in that project in order to add support for macros to the Play Json. Following the same pattern, you can easily use almost any other Json library like this.

Lets look what it gives you without any reflections.. Lets start with something simple:
    scala> import info.akshaal.json.play._
    import info.akshaal.json.play._

    scala> import play.api.libs.json._
    import play.api.libs.json._

    scala> case class Simple(str: String, num: Int = new Random().nextInt())
    defined class Simple

    scala> implicit val simpleJsFactory = factory[Simple]('fromJson)
    simpleJsFactory: (Symbol => Option[play.api.libs.json.JsValue]) => Simple = <function1>

    scala> implicit val simpleJsFields = allFields[Simple]('jsonate)
    simpleJsFields: List[info.akshaal.clazz.Field[Simple,play.api.libs.json.JsValue,None.type]] = List(Field(str,<function1>,None), Field(num,<function1>,None))
We can serialize and then deserialize:
    scala> val obj = new Simple("123", 5)
    obj: Simple = Simple(123,5)

    scala> val objJs = Json.toJson(obj)
    objJs: play.api.libs.json.JsValue = {"str":"123","num":5}

    scala> val obj2 = Json.fromJson[Simple](objJs)
    obj2: Simple = Simple(123,5)
That was easy. Now lets try to leverage default value feature of the case class:
    scala> val halfObjJs = Json.parse(""" {"str":"test"} """)
    halfObjJs: play.api.libs.json.JsValue = {"str":"test"}

    scala> val halfObj = Json.fromJson[Simple](halfObjJs)
    halfObj: Simple = Simple(test,-813663852)

    scala> val halfObj = Json.fromJson[Simple](halfObjJs)
    halfObj: Simple = Simple(test,-948806581)

    scala> val halfObj = Json.fromJson[Simple](halfObjJs)
    halfObj: Simple = Simple(test,428745442)
Note that we parsed halfObjJs three times. And 'num' was always different. That is because value for 'num' is missing in the halfObjJs and so the expression for default value was used (which is Random().nextInt, see defintion of Simple class). How about parametrized classes? Lets try:
    scala> case class Event[T](kind: String, payloads: T)
    defined class Event

    scala> implicit def eventJsFields[T: Writes] = allFields[Event[T]]('jsonate)
    eventJsFields: [T](implicit evidence$1: play.api.libs.json.Writes[T])List[info.akshaal.clazz.Field[Event[T],play.api.libs.json.JsValue,None.type]]

    scala> implicit def eventJsFactory[T: Reads] = factory[Event[T]]('fromJson)
    eventJsFactory: [T](implicit evidence$1: play.api.libs.json.Reads[T])(Symbol => Option[play.api.libs.json.JsValue]) => Event[T]
And test:
    scala> val event = Event(kind = "strange", payloads = obj)
    event: Event[Simple] = Event(strange,Simple(123,5))

    scala> val eventJs = Json.toJson(event)
    eventJs: play.api.libs.json.JsValue = {"kind":"strange","payloads":{"str":"123","num":5}}

    scala> val event3 = Json.fromJson[Event[Simple]](eventJs)
    event3: Event[Simple] = Event(strange,Simple(123,5))
In the real world it is unlikely that you will use completely unrestricted types (like T in the example above) in your case classes with json. It is more likely that there will be a (sealed) trait and some set of its subtypes. Lets see how it works. First, we will define some more case classes:
    sealed trait Message
    case class QuitMessage(msg: String) extends Message
    case class Heartbeat(id: Int) extends Message

    case class Messages(userId: Int, messages: List[Message] = List.empty)
Now, lets define how to serialize our new case classes:
    implicit def messageWrites = matchingWrites[Message] {
        case m: QuitMessage => quitMessageJsFields.toWrites.writes(m)
        case m: Heartbeat   => heartbeatJsFields.toWrites.writes(m)
    }

    implicit def quitMessageJsFields = allFields[QuitMessage]('jsonate)
    implicit def heartbeatJsFields = allFields[Heartbeat]('jsonate)
    implicit def messagesJsFields = allFields[Messages]('jsonate)
And if deserialization is needed, lets define how to do it:
    implicit def quitMessageJsFactory = factory[QuitMessage]('fromJson)
    implicit def heartbeatJsFactory = factory[Heartbeat]('fromJson)
    implicit def messagesJsFactory = factory[Messages]('fromJson)

    implicit def messageReads: Reads[Message] =
        predicatedReads[Message](
            jsHas('msg) -> quitMessageJsFactory,
            jsHas('id) -> heartbeatJsFactory
        )
Finally, lets try it:
    scala> val event5 =
              Event(kind = "t1",
                    payloads = Messages(userId = 4, messages = List(Heartbeat(5), QuitMessage("bye!"))))
    event5: Event[Messages] = Event(t1,Messages(4,List(Heartbeat(5), QuitMessage(bye!))))

    scala> val event5Js = Json.toJson(event5)
    event5Js: play.api.libs.json.JsValue = {"kind":"t1","payloads":{"userId":4,"messages":[{"id":5},{"msg":"bye!"}]}}
Pay attention, that Json for QuitMessage and Heartbeat classes has no type information or anything! It's just plain json with domain fields only. That's why messageReads has those jsHas occurances in its implementation, that is a little help for identifying which json object is what subtype of Message. Lets see that reading from json still works:
    scala> val event6 = Json.fromJson[Event[Messages]](event5Js)
    even6: Event[Messages] = Event(t1,Messages(4,List(Heartbeat(5), QuitMessage(bye!))))
Actually there is another way to do the same. You can inject an extra field into json object when serializing one of subtypes of an abstract type and use it as a guidance for reconstructing objects from json. It is really easy. Lets modify messageReads and messageWrites like this:
    implicit def messageWrites = matchingWrites[Message] {
        case m: QuitMessage => quitMessageJsFields.extra('type -> 'quit).toWrites.writes(m)
        case m: Heartbeat   => heartbeatJsFields.extra('type -> 'heart).toWrites.writes(m)
    }

    implicit def messageReads: Reads[Message] =
        predicatedReads[Message](
            jsHas('type -> 'quit)  -> quitMessageJsFactory,
            jsHas('type -> 'heart) -> heartbeatJsFactory
        )
Now we test the new implementation.
    scala> val messages = List(Heartbeat(1), QuitMessage("Hello"), Heartbeat(99))
    messages: List[Product with Serializable with Message] = List(Heartbeat(1), QuitMessage(Hello), Heartbeat(99))

    scala> val messagesJs = Json.toJson(messages)
    messagesJs: play.api.libs.json.JsValue = [{"type":"heart","id":1},{"type":"quit","msg":"Hello"},{"type":"heart","id":99}]

    scala> val messages2 = Json.fromJson[List[Message]](messagesJs)
    messages2: List[Message] = List(Heartbeat(1), QuitMessage(Hello), Heartbeat(99))
That's not all. What if there is an information in a case class you don't want to reveal in JSON? Consider the following case class. I will annotate fields that are safe to export by Ok annotation:
    @annotation.meta.getter
    class Ok extends annotation.StaticAnnotation

    case class User(@Ok login: String,
                    @Ok fullName: String,
                    @Ok messages: Int = 0,
                    passwordHash: Option[String] = None)
Now it's quite natural to define json fields like this:
    implicit val userJsFields = annotatedFields[User, Ok]('jsonate)
Let see it in action:
    scala>
    |       val user = User(login = "akshaal",
    |                       fullName = "Evgeny Chukreev",
    |                       messages = 10,
    |                       passwordHash = Some("4d18758602c08243d7c08f8c9e4463b0"))
    user: User = User(akshaal,Evgeny Chukreev,10,Some(4d18758602c08243d7c08f8c9e4463b0))

    scala> val userJs = Json.toJson(user)
    userJs: play.api.libs.json.JsValue = {"login":"akshaal","fullName":"Evgeny Chukreev","messages":10}
It works.. But you can do more. Recall (if you looked at the implementation) that jsonate function was defined like this:
    def jsonate[T: Writes](t: T, args: Any): JsValue = Json.toJson(t)
The function is used to make a json value out of field's value. It is applied to each field. So you can define your our function that does post-processing (pre-processing?) and use it with a macro. The second argument (args) might be a parameter set given to annotation, this gives you even more power for writing complex json serialization easily.

And not only JSON. Using the same approach you can (de)serialize object from/to XML...

Pros of akmacros-json

  • Domain classes are separated from any notion of JSON
  • Full control over serialization/deserialization
  • Easy to use
  • Easy to extend or implement your own
  • No runtime reflections used

Cons of akmacros-json

  • Depends on Jerkson which is officially unavailable for Scala 2.10 (you need to build it from my fork (in order to build it you will need also this forked project))
  • Includes a copy of Play Json
  • Scala 2.10-M7 has many bugs related to implicits, value classes... so implementation as you might have noticed is not perfect in terms of performance (defs are used instead of vals)
I hope things will change really soon with the release of Scala 2.10.

About general purpose macros

This tiny macros addition is built on top of Play JSON (which is built on top of Jerkson (which is built on top of Jackson)) and akmacros (which doesn't have dependencies). Checkout 78 lines long implementation here. See https://github.com/akshaal/akmacros for a bit more information about using akmacros with sbt.

Wednesday, September 5, 2012

Easily implementing Json serialization for Play using macro. Function by symbol (lisp-like)l

I liked the idea of using fields macro in scala so much so I created a dedicated project for this macro to start reusing it in the different projects I did. Here it is.

Few words about implementation of this macro

The code has been reworked and now it is possible to transform field value using a supplied function. It means that Field and Fields types, and fields macro signatures are changed. In addition, there is a new convenient macro called allFields. I.e.:
    case class Field[-I <: AnyRef, +R, +A <: Product](name: String, get: I => R, args: A)
    type Fields[-I <: AnyRef, +R, +A <: Product] = List[Field[I, R, A]]
    def fields[Ann, I <: AnyRef, R, Args <: Product](apply: Symbol) = macro fieldsImpl[Ann, I, R, Args]
    def allFields[I <: AnyRef, R](apply: Symbol) = macro fieldsImpl[Any, I, R, None.type]
allFields is the new function that lists all public value members enclosed in the given type regardless of annotations. R in a Field(s) type stands for RETURN and represents type of a value returned by the field getter transformed using the supplied function. Value transformer function is passed into the macro using its symbol. You might wonder why aren't we just using something like f : T => R forSome { type T } ? That's because you can't pass the following function that way:
    def trans[T : TypeClass](x : T) : Int = ???
(which is just a sweet way of saving some keystrokes by not writing this:)
    def trans[T](x : T)(implicit tc : TypeClass[T]) : Int = ???
i.e. you can't pass function with TWO parameters (one of which is implicit parameter) where a ONE parameter function is expected. That is quite obvious but anyway.. So we use Symbol (in the spirit of Lisp). As you might guess by looking at the snippet below, it is expected that the symbol is constructed directly and not passed by reference:
        val applyFunName =
            apply.tree match {
                case Apply(_, List(Literal(Constant(s)))) => s.toString
                case _ =>
                    c.abort(apply.tree.pos,
                        "fields macro is expected to be used with symbol literal like 'nothing or 'myFunction")
            }
Almost everything in the macro implementation remains more-less same, except part that constructs expression for getting field value out of object. Now it looks like this:
                val applyFunTree = c.parse(applyFunName)
                val getFunArgTree = ValDef(Modifiers(), newTermName("x"), TypeTree(instanceT), EmptyTree)
                
                val getFunBodyTree =
                    treeBuild.mkMethodCall(applyFunTree,
                        List(Select(Ident(newTermName("x")), newTermName(name)),
                             argsTree))
getFunBodyTree illustrates what signature is really expected for the transformer function: in addition to field value, all arguments of the annotation are passed into the function (or None if no annotation used or annotation has no arguments). For example, you can't use Predef.identity function, instead, you should use valueIdentity which is (already) defined like this:
def valueIdentity[X] (value : X, annotationArgs : Any) : X = value
Having annotation arguments provided for the currently processing field gives you possibility for further customization of how the value is transformed. Now lets do an example.

Real-world example

Suppose you want to serialize your custom classes into JSON with no boilerplate code what so ever. That is how you can do it with this only (general-purpose) macro. Lets define some generic Writes typeclase provider:
    implicit def writesForFields[T <: AnyRef](implicit fields: clazz.Fields[T, JsValue, _]): Writes[T] = {
        new Writes[T] {
            def writes(t: T): JsValue = {
                JsObject(fields map {
                    (field: clazz.Field[T, JsValue, _]) =>
                        field.name -> field.get(t: T)
                })
            }
        }
    }
The function shown above implicitly creates Writes for any type T which has an implicit instance of type Fields[T, JsValue, _] (read it like "List of fields of class T along with function to get value of type JsValue for each field"). Now lets define the transformer function, it will be used for serialization of field values:
def asJsValue[T : Writes](v : T, annArgs : Any) : JsValue = Json.toJson(v)
That was the only code needed to bootstrap your mini-serialization framework. Now you can use it. Lets assume you have declarations:
    case class JquerySocketEvent[T](id: Int, data: T, `type`: String = "message", reply: Boolean = false)
    case class ChatMessage(user: String, text: String)
    
    implicit def jquerySocketEventJsFields[T: Writes] = clazz.allFields[JquerySocketEvent[T], JsValue]('asJsValue)
    implicit val chatMessageJsFields = clazz.allFields[ChatMessage, JsValue]('asJsValue)
That's it. ... some fun:
        val event = JquerySocketEvent(id = 1, data = ChatMessage("Fluttershy", "Yay!"))
        println (Json.toJson(event))
... prints:
    {"id":1,"data":{"user":"Fluttershy","text":"Yay!"},"type":"message","reply":false}
That was easy enough. Feel free to use it, re-implement it or implement a more powerful stuff. Macros FTW!

Saturday, August 25, 2012

Kiama & macro

Here is a quick and dirty example of using kiama along with the fields macro.
import annotated.{ Field => AnnField, Fields => AnnFields }

object pp {
    private object kpp extends org.kiama.output.PrettyPrinter
    import kpp._

    private def anyToDoc(any : Any) : Doc =
        any match {
            case song : Song        => annotatedToDoc("Song", song, songFields)
            case artist : Artist    => annotatedToDoc("Artist", artist, artistFields)
            case comp : Compilation => annotatedToDoc("Compilation", comp, compilationFields)
            case job : UploaderJob  => annotatedToDoc("Job", job, uploaderJobFields)

            case map : Map[_, _] =>
                list(map.iterator.toList,
                    prefix = "Map",
                    elemToDoc = {
                        (pair : (Any, Any)) =>
                            anyToDoc(pair._1) <> " -> " <> nest(anyToDoc(pair._2))
                    })

            case seq : Seq[_] =>
                list(seq.toList, prefix = "Sequence", elemToDoc = anyToDoc)

            case _ => value(any)
        }

    private def annotatedToDoc[T <: AnyRef](name : String, t : T, fields : AnnFields[T, FieldArgs]) : Doc = {
        list(fields,
            prefix = name,
            elemToDoc = {
                (f : AnnField[T, FieldArgs]) =>
                    f.name <> " = " <> anyToDoc(f.get(t))
            })
    }

    def apply(any : Any) : String = pretty(anyToDoc(any))
}
pp (uploaderJob) easily prints data like this:
Job(
    artistMap = Map(
        KeyRef(1x) -> Artist(
                handle = Tester2,
                marks = Set(),
                id = None,
                name = None,
                website = None,
                country = None,
                location = None),
        KeyRef(2u) -> Artist(
                handle = Tester,
                marks = Set(),
                id = None,
                name = None,
                website = None,
                country = None,
                location = None)),
    songMap = Map(
        KeyRef(6t) -> Song(
                relativePath = Path(test7.mp3),
                title = test7,
                artistRefs = Set(KeyRef(2u)),
                tags = Set(),
                id = None,
                sourceId = None,
                mixSongId = None,
                year = Some(1999)),
        KeyRef(5c) -> Song(
                relativePath = Path(test6.mp3),
                title = test6,
                artistRefs = Set(KeyRef(2u)),
                tags = Set(),
                id = None,
                sourceId = None,
                mixSongId = None,
                year = Some(1999)),
        KeyRef(9i) -> Song(
                relativePath = Path(test3.mp3),
                title = test3,
                artistRefs = Set(KeyRef(1x)),
                tags = Set(),
                id = None,
                sourceId = Some(4),
                mixSongId = None,
                year = None),
        KeyRef(3w) -> Song(
                relativePath = Path(test5.mp3),
                title = test5,
                artistRefs = Set(KeyRef(2u)),
                tags = Set(),
                id = None,
                sourceId = None,
                mixSongId = None,
                year = Some(1999)),
        KeyRef(7l) -> Song(
                relativePath = Path(test.mp3),
                title = test,
                artistRefs = Set(),
                tags = Set(),
                id = None,
                sourceId = None,
                mixSongId = None,
                year = None),
        KeyRef(8f) -> Song(
                relativePath = Path(test2.mp3),
                title = test2,
                artistRefs = Set(),
                tags = Set(),
                id = None,
                sourceId = None,
                mixSongId = None,
                year = None),
        KeyRef(4p) -> Song(
                relativePath = Path(test4.mp3),
                title = test4,
                artistRefs = Set(KeyRef(2u)),
                tags = Set(),
                id = None,
                sourceId = None,
                mixSongId = None,
                year = Some(1999)),
    compilationMap = Map(
        KeyRef(11x) -> Compilation(
                title = Compo,
                songRefs = Sequence(KeyRef(6t)),
                marks = Set(),
                year = Some(1999),
                cdOrSide = Some(B)),
        KeyRef(10g) -> Compilation(
                title = Compo,
                songRefs = Sequence(KeyRef(3w), KeyRef(4p), KeyRef(5c)),
                marks = Set(),
                year = Some(1999),
                cdOrSide = Some(A))))

Saturday, August 18, 2012

Scala 2.10: annotated fields macro

Here is a short example of how one can leverage SIP-16 introduced in Scala-2.10.
(The source code you will find below is expected to be compiled on Scala 2.10-M7. Note, that -M6 provides a slightly different set of API for macro.)

Lets define a macro that makes it possible to traverse value fields of a (case) class. First, we import what we will use:
import language.experimental.macros
import scala.reflect.macros.Context
import scala.annotation.Annotation
The macros, we are implementing, will be located in 'annotated' object since Scala allows usage of type aliases inside object (unlike package namespace).
object annotated {
Any field belongs to a class (denoted as I). An annotated field might have useful information given by arguments on annotation. Type of the annotation arguments is denoted as A. Here is the definition of the Field class:
    /**
     * An object of this class represents an annotated field.
     * @tparam I type of class the field belongs to
     * @tparam A type of annotation arguments (TupleX or None)
     * @param name name of the field
     * @param get function that returns field value of an instance given as argument to the function
     * @param args list of arguments to the annotation found on the field
     */
    case class Field[I <: AnyRef, A <: Product](name : String, get : I => Any, args : A)
Here is the type alias to save some typing:
    /**
     * List of fields belonging to the given type.
     * @tparam I Owner of fields
     * @tparam A type of annotation arguments (TupleX or None)
     */
    type Fields[I <: AnyRef, A <: Product] = List[Field[I, A]]
That is how our macro is supposed to be seen by developers (i.e. it is supposed to be seen as an ordinary function):
    /**
     * Macro which inspects class 'I' and returns a list of fields annotated with annotation 'Ann'.
     * @tparam Ann search for field with this annotation
     * @tparam Args type of arguments in the annotation (TupleX or None)
     * @tparam I type of class to scan for annotated fields
     */
    def fields[Ann <: Annotation, Args <: Product, I <: AnyRef] = macro fieldsImpl[Ann, Args, I]
Finally, here is the implementation of the macro itself. The implementation is called by the scala compiler whenever it sees 'fields' macro:
    /**
     * Implementation of the fields macro.
     */
    def fieldsImpl[AnnTT <: Annotation : c.AbsTypeTag,
                   Args <: Product : c.AbsTypeTag,
                   ITT <: AnyRef : c.AbsTypeTag](c : Context) : c.Expr[Fields[ITT, Args]] = {
("Args <: Product : c.AbsTypeTag" means "type Args is a subtype of type Product and there is an implicit value of type c.AbsTypeTag[Args]")
Note that here and further below we use types (like AbsTypeTag) which are from the context 'c'. That is a compilation context of the application which the scala compiler will construct for the source code where the macro invocation is faced (not exactly but..).

Now lets import types and values (like Select, Ident, newTermName) from the universe of the application the macro is currently used in:
        import c.universe._
Lets materialize some types as objects for further manipulation:
        val instanceT = implicitly[c.AbsTypeTag[ITT]].tpe
        val annT = implicitly[c.AbsTypeTag[AnnTT]].tpe
Now, some real action: get annotated fields. Note that hasAnnotation doesn't work for a reason I don't know...
        val annSymbol = annT.typeSymbol
        val fields = instanceT.members filter (member => member.getAnnotations.exists(_.atp == annT))
It is convenient to have a helper function. This function will fold given expression sequence into a new expression that creates List of expressions at runtime ;-)
        def foldIntoListExpr[T : c.AbsTypeTag](exprs : Iterable[c.Expr[T]]) : c.Expr[List[T]] =
            exprs.foldLeft(reify { Nil : List[T] }) {
                (accumExpr, expr) =>
                    reify { expr.splice :: accumExpr.splice }
            }
For each field, construct expression that will instantiate Field object at runtime:
        val fieldExprs =
            for (field <- fields) yield {
                val argTrees = field.getAnnotations.find(_.atp == annT).get.args
                val name = field.name.toString.trim // Why is there a space at the end of field name?!
                val nameExpr = c literal name

                // Construct arguments list expression
                val argsExpr =
                    if (argTrees.isEmpty) {
                        c.Expr [Args] (Select(Ident(newTermName("scala")), newTermName("None")))
                    } else {
                        val tupleConstTree = Select(Select(Ident(newTermName ("scala")),
                                                           newTermName(s"Tuple${argTrees.size}")),
                                                    newTermName("apply"))
                        c.Expr [Args] (Apply (tupleConstTree, argTrees))
                    }
                    
                // Construct expression (x : $I) => x.$name
                val getFunArgTree = ValDef(Modifiers(), newTermName("x"), TypeTree(instanceT), EmptyTree)
                val getFunBodyTree = Select(Ident(newTermName("x")), newTermName(name))
                val getFunExpr = c.Expr[ITT => Any](Function(List(getFunArgTree), getFunBodyTree))
                
                reify {
                    Field[ITT, Args](name = nameExpr.splice, get = getFunExpr.splice, args = argsExpr.splice)
                }
            }
By this moment, value fieldExprs will contain something like
List (
   reify {Field ('field1', (x => x.field1), (..)},
   reify {Field ('field2', (x => x.field2), (..)}
)
(where (..) are arguments of annotaiton on that field)
Now we have to lift List construction into expression and we're done!
       // Construct expression list like field1 :: field2 :: Field3 ... :: Nil
        foldIntoListExpr(fieldExprs)
    }
}

And finally lets have some fun. Lets test it in REPL! (beware that scala macros are supposed to be compiled before they are used)
scala> type FormatFun = Any => Any
defined type alias FormatFun

scala> type PrettyArgs = (Option[String], FormatFun)
defined type alias PrettyArgs

scala> class Pretty(aka : Option[String] = None, format : FormatFun = identity) extends annotation.StaticAnnotation
defined class Pretty
scala> :paste
// Entering paste mode (ctrl-D to finish)

def pp[X <: AnyRef](fields : annotated.Fields[X, PrettyArgs])(x : X) = {
    fields map {
        case annotated.Field(fieldName, get, (akaOpt, fmtFun)) =>
            val name = fieldName.replaceAll("([A-Z][a-z]+)", " $1").toLowerCase.capitalize
            val aka = akaOpt map (" (aka " + _ + ")") getOrElse ""
            val value = fmtFun(get(x))
            s"$name$aka: $value"
    } mkString "\n"
}
        
// Exiting paste mode, now interpreting.

pp: [X <: AnyRef](fields: info.akshaal.radio.uploader.annotated.Fields[X,(Option[String], Any => Any)])(x: X)String

Still no macro was used. Now here it comes. First, we define case class. Next, we gather annotated fields in the definition of personPrettyFields. When you run it in REPL, it is quite important to use :paste, otherwise annotation of the case class will be lost because subtypes of StaticAnnotation are visible during compilation only (REPL calls a new scala compiler for each expression reusing binary classes compiled during previous steps). So:
scala> :paste
// Entering paste mode (ctrl-D to finish)

case class Person(
    id : Int,
    @Pretty(aka = Some("nickname")) name : String,
    @Pretty firstName : String,
    @Pretty(None, format = _.toString.toUpperCase) secondName : String,
    @Pretty(None, format = { case x : Option[_] => x getOrElse "" }) twitter : Option[String])
val personPrettyFields = annotated.fields[Pretty, PrettyArgs, Person]
        
// Exiting paste mode, now interpreting.

defined class Person
personPrettyFields: List[info.akshaal.radio.uploader.annotated.Field[Person,(Option[String], Any => Any)]] =
    List(Field(name,<function1>,(Some(nickname),<function1>)),
         Field(firstName,<function1>,(None,<function1>)),
         Field(secondName,<function1>,(None,<function1>)),
         Field(twitter,<function1>,(None,<function1>)))
(I've aligned the output of REPL a bit..)
Lets check field names:
scala> personPrettyFields map (field => field.name)
res0: List[String] = List(name, firstName, secondName, twitter)
Here are getters of each field:
scala> personPrettyFields map (_.get)
res1: List[Person => Any] = List(<function1>, <function1>, <function1>, <function1>)
Now, lets create a Person object:
scala> val person1 = Person(1, "akshaal", "Evgeny", "Chukreev", Some("https://twitter.com/Akshaal"))
person1: Person = Person(1,akshaal,Evgeny,Chukreev,Some(https://twitter.com/Akshaal))
... and a value for each field of this person:
scala> personPrettyFields map (_.get (person1))
res2: List[Any] = List(akshaal, Evgeny, Chukreev, Some(https://twitter.com/Akshaal))
Some more objects for more fun:
scala> val person2 = Person(2, "BillGates", "Bill", "Gates", Some("https://twitter.com/BillGates"))
person2: Person = Person(2,BillGates,Bill,Gates,Some(https://twitter.com/BillGates))

scala> val persons = List(person1, person2)
persons: List[Person] =
    List(Person(1,akshaal,Evgeny,Chukreev,Some(https://twitter.com/Akshaal)),
         Person(2,BillGates,Bill,Gates,Some(https://twitter.com/BillGates)))

scala> val ppPerson = pp(personPrettyFields) _
ppPerson: Person => String = <function1>         
And finally:
scala> persons map ppPerson mkString "\n----------------------------\n"
res5: String =

Name (aka nickname): akshaal
First name: Evgeny
Second name: CHUKREEV
Twitter: https://twitter.com/Akshaal
----------------------------
Name (aka nickname): BillGates
First name: Bill
Second name: GATES
Twitter: https://twitter.com/BillGates

That's all ;-) You will find complete source code along with specs2 specification (with one more example) on the gist: https://gist.github.com/3388753

No animals were killed.

No types were casted.

No reflections were used.

Wednesday, January 20, 2010

nut & ippon

It was tricky to make nut 2.4.1 work with IPPON Power PRO 1000 using USB interface.

Any driver (blazer_usb, megatec_usb) failed with the message:
Can't claim USB device [xxxx:yyyy]: could not detach kernel driver from interface 0: Operation not permitted

It turned out, that nut came with the broken configuration for udev 150 (on debian). To fix the problem, I commented out all lines in /etc/udev/rules.d/52-nut-usbups.rules, and created a new file with the name /etc/udev/rules.d/46-ippon.rules and the following content:

SUBSYSTEM=="usb", ATTR{idVendor}=="0665", ATTR{idProduct}=="5161", MODE="0664", GROUP="nut"

This helped. Required product id and vendor id are discovered from the output of lsusb program.

PS. The following alias is very useful when configuring something with udev:
alias udevinfo="udevadm info -a -n"
and it can be used like:
udevinfo /dev/sda

Thursday, November 26, 2009

Upgrade of Firefox

After apt-get dist-upgrade I got this "Could not initialize the browser's security component. The most likely cause is problems with files in your browser's profile directory. Please check that this directory has no read/write restrictions and your hard disk is not full or close to full. It is recommended that you exit the browser and fix the problem. If you continue to use this browser session, you might see incorrect browser behavior when accessing security features."

I tried many possible solutions discovered in internet, but the only thing helped was removing of secmod.db file from my profile directory of firefox. I have no idea what this file is for, but it works. If you are having the same problem, backup your ~/.mozilla directory first before trying this or other solution which involves removing of files.