Thoughts on Devaamo Summit 2012

I visited this year's edition of Devaamo Summit 2012 around a week ago. Devaamo Summit is an event focusing on various open themes held at Tampere (Manchester of Finland). There were two tracks this year: open web & data and open mobile & hardware.

This was the third time it was organized. As far as I understood it was previously focused on MeeGo but as you might know, things didn't go so well with that project. Even this time there were some issues (Nokia just laid of 10k people, 3.7k of which in Finland, a big deal locally).

The event lasted for two days. The first day, really short one, had some lightning talks. The second day contained the actual beef. The topics of the presentations were well chosen and there was probably something interesting available for everyone. In addition there were a few booths around and a hacking challenge even.

Somewhere near Tampella factories

Thoughts on Lightning Talks

I didn't take for any specific notes for the lightning talks so going by memory here. The format was standard 5 min (3 mins for presentation, 2 mins for questions). Sadly this was not enforced strongly enough so some talks kept on meandering around. Hopefully they can deal with this better in the next year's event.

Topicwise it was a mixed bag. There were some business related talks and more technical ones. I'm going to elaborate a bit on some of the technical ones since I found those most interesting:

• Lauri Paimen demonstrated his implementation of QML in the web browser. QML is a declarative language familiar from the Qt framework. It looks a bit like JSON but allows you to describe some basic logic. Note that Paimen's QMLWEB is still in quite an early phase (proof of a concept). You might want to investigate the presentation slides (implemented using QML of course) to get into it better.
• Otto Kekäläinen spoke about VALO-CD. Essentially it's a CD (do'h) containing a collection of open source software. It has been meant to function as a gateway for people stuck on Windows. Get them use the apps first and hook them on Linux later. :)
• Tero Piirainen introduced ITS Factory and Tampere's plans of opening some traffic data. I really hope this catches on in Finland.
• Antti-Jussi Kovalainen discussed his Beatstream project. Essentially that is a service that makes it possible for you to stream audio from your home computer to anywhere using a web browser. The source is available at GitHub of course.

Thoughts on Presentations

Given there were two tracks I had to make some tough choices about which presentations to see. I focused mainly on the open web & data track, though I saw a few on the other track as well. Overall the quality of the presentations was good.

I have to admit some felt a bit underwhelming given they were clearly aimed for people not so aware of open solutions. I also felt that sometimes the thirty minute slot given for each talk was not enough for some. It is simply not possible to delve very deep in such a short frame of time.

Acceptance Test Driven Development for Web Apps

Eemeli Kantola of Futurice spoke about Acceptance Test Driven Development for Web Apps. The presentation remained on quite a high level. It seems to me ATDD might fit well in an environment where you have a good understanding of various requirements. Mapping these to tests using ATDD feels like a natural approach. That way you get something less vulnerable to regressions and will be able to maintain a sustainable pace.

This was one of those talks that were hindered by the fixed 30 min slot. It would have been really interesting to see how it works out in practice. Fortunately Eemeli has prepared a demo project that ought to give some idea.

Best Practices for HTML5 Servers

Otto Kekäläinen of Seravo went through some Best Practices for HTML5 Servers. Overall the talk was somewhat oriented towards beginners. If you've ever dealt with servers, you probably know this stuff already. It would have been somewhat interesting to have some kind of comparison of old skool servers (self-hosted) and cloud based solutions (Heroku, OpenShift) in the presentation.

Decoupling Content Management with CreateJS

Henri Bergius of Nemein discussed about Decoupling Content Management with Create.JS. He aims to change the way we think about CMS. Instead of treating it as a monolithic entity, he proposed a way to decouple it. Essentially he splits the whole in three parts: web editing tool (content manipulation, publishing), web framework (business logic, serving the content), content repository (database and related ops). The main advantage of this is that you can use the CMS you want with the UI you want.

I believe this sort of approach might have some kind of a future ahead of it. It definitely makes sense architecture-wise. Whether it works out in practice and gains some popularity remains to be seen.

This is another presentation that would have benefitted from a longer slot (60 mins even). The slides I linked to seem to be from a longer talk.

Trends and Visions in 3D Printing Community - Survey Results

Jarkko Moilanen's "Trends and Visions in 3D Printing Community - Survey Results" felt like a blast from the academic past, and to be honest that's what it was. I guess the main thing I got out of this was that an average person dabbling with 3D printing is a European 35 old male with university background. Oh, and the community is somewhat fragmented. The people dealing with 3D printing identify with the maker movement.

Based on the results it seems 3D printing is on its way to further commercial usage. Currently the main problems on hobbyist side seem to be accuracy, speed and material limitations (pretty much just plastic for now). I think B9Creator might solve the first two. That makes the current consumer level printers look like ancient technology (consider the leap between NES and SNES or PS and PS2).

Intro to 3D Parametric Design

Randall Arnold of Tribal Method gave an Intro to 3D Parametric Design. Even though the presentation was interesting it felt a bit underwhelming to me. Sure, it gave a nice idea of the paradigm. Still, it would have been nice to get a bit more into the detail.

Parametric modeling is just one of the basic ways you can use to produce 3D models. It is especially suitable for producing objects to print given it's easy to end up with something that's manifold (ie. the object doesn't have "holes"). Boolean and formative (mesh) modeling are commonly used as well. Each approach has its sides.

In parametric modeling you describe your object as a set of operations. In essence you are constructing a graph of operations. In addition you will define certain kind of constants and parameters. Suppose you are modeling a hex bolt this way. You might want to treat the dimensions of the upper part (the one with hexagon) as constants while the height of the screwy part could vary. This allows you to produce a bolt with given height quite easily.

I prefer mesh modeling myself. Of course it isn't mathematically as pure as boolean or parametric approaches. You can go from boolean or parametric model to mesh easily but not the other way. There are some hacks (subdivision surfaces, modifiers etc.) that make the situation fuzzier, though.

This is the kind of stuff I would've like to see in the presentation. As it was I really didn't learn a lot from it. Again, 30 minutes was way too short a slot to go through comprehensively a topic such as this.

Going into Cloud - New Conceptual Model for Personal Information

Jussi Alanen's presentation underlined the importance of having a proper access to personal information scattered around the cloud. According to him (and Gartner) "personal cloud" is going to replace PC by 2014. These guys are taking the prediction seriously and plan to do something about it.

Currently the data has been scattered all around various services (Facebook, Twitter, Google, Flickr, you name it). Just having access to the data is not enough. A sensible way to organize it is required.

If I understood correctly they aim to combine data from multiple sources using metadata. This way you could construct a certain kind of view that would give you access to data related to some specific pivot. Consider that you participate in some event (like Devaamo Summit). You might want to have access just to photos and contacts related to it.

This kind of scheme might make it possible to implement new kind of services. There are plenty of mashups available already. This approach would probably take it to the next level.

It will be interesting to see what they will be able to achieve based on this premise. If you are interested in this sort of stuff, you might want to check out their site named "Bring my information home!".

Thoughts on Other Presentations

I caught a glance of Pekka Sarkola's "OpenStreetMap and Free Open Geodata Are Great Opportunities". Something interesting happened a while ago at May. Most of Finnish geodata available was published with a permissive license. This is a huge deal. It wouldn't surprise me if we saw various enterprises tapping into the data.

Unfortunately "Hack Avoin Ministeriö" by Aleksi Rossi was cancelled at the last moment (the speaker didn't show up). Avoin ministeriö (open ministry) aims to apparently make it easier for the citizens to take part in the legislation process. It seems like a step to the right direction.

Conclusion

Overall Devaamo Summit was a nice little event. If it is organized next year and I don't have any other appointments I'll likely visit there again. Thanks to the organizers and speakers. Special mention goes to Futurice and their stand for the swag. :)

Developing a REST API And Client Using Node

The web is full of APIs. These days mashups are getting more and more popular. People aggregate information from various sources to produce new, sometimes interesting, services. REST (Representational State Transfer) provides one nice way to come up with an API that is fairly easy to consume.

In this post I'll discuss REST briefly and introduce you to a few tools I developed in order to make it easier to implement and consume REST APIs using Node. As an API is somewhat useless without some kind of a database behind it, I'll discuss MongoDB (or rather Mongoose) very briefly as well.

Introduction to REST

So what is REST? You can probably find a lot of definitions out there. For me it's all about resources and HTTP verbs POST, GET, PUT and DELETE. These verbs map to CRUD (Create, Read, Update, Delete) familiar from persistent storage. You can think it as a web specific way of doing the same thing.

This mapping means we can model our backend behind a REST API. This decoupling is quite powerful as it allows us to use whatever technology we like on the backend and frontend side. In fact we could write multiple different frontends hooking into our backend. The frontends could even mash data from multiple sources. Compared to the traditional "lets lump it all together and write some APIs later" approach this seems like a more novel way to go.

REST doesn't specify in any manner how to deal with auth. There are multiple possible solutions for this particular problem. In this post I will use a combination of API key over SSL since it's quite simple to set up and fits my purposes for now. OAuth and Amazon style HMAC are popular alternatives too.

Anatomy of a REST API

I really like the RESTful API Design slides. I think they summarize quite well what REST is about and how you might go and and design your API. Consider we're writing a TODO application. We might design a Todo model with fields such as these:

• Name - String, required
• Priority - Integer within [0, 5], defaults to zero
• Description - String, required
• Status - String enum (either "Waiting", "Doing" or "Done"), defaults to "Waiting"

Our REST API for this very simple schema can be modeled simply as a route /api/v1/todo. The following list describes the operations we can provide through the API:

• POST /todo - Creates a new Todo resource and returns it
• GET /todo - Returns existing Todo resources
• PUT /todo - Not allowed (403 error)
• DELETE /todo - Not allowed (403 error)

Those operations function on collection level so to speak. The following operate on an individual item.

• POST /todo/ - Not allowed (403 error)
• GET /todo/ - Returns resource and returns it if found, else 404 error
• PUT /todo/ - Updates resource and returns it if found, else 404 error 
• DELETE /todo/ - Deletes resource and returns empty if found, else 404 error

"RESTful API Design" slides describe some additional functionality. For instance it might make sense to constrain fields returned (ie. fields=name,priority), match by some property (ie. name=some+todo), count (ie. /count) or perhaps use pagination (ie. limit=25&offset=50).

rest-sugar, mongoose-sugar, rest-sugar-client

In fact a utility of mine, rest-sugar, does what I just described and then some. It has been designed to work with mongoose-sugar, another package of mine. There is a third package, rest-sugar-client, that makes it easy to consume the API constructed by rest-sugar.

So how this all works? What's up with all the sugar? Not so sure about sugar. Just popped into mind, really. The whole point of these tools is to make it easy to generate a REST API based on the given database schemas and provide a high level API for the client.

In this case the database schema could be defined using mongoose-sugar. That works as a light wrapper to Mongoose that in turn wraps MongoDB. rest-sugar hooks into this definition and queries provided by mongoose-sugar and constructs a REST API based on that. Finally rest-sugar-client creates the client side API based on the metadata available on the REST API.

mongoose-sugar

mongoose-sugar wraps Mongoose and makes schemas behave in a way more appropriate for me. For instance I made sure all my schemas are set as "strict" by default. This means they will get validated on database level. I also made sure certain extra fields (ie. deleted flag and created date) get attached to each schema. I added that date information there just for convenience. As I prefer to deal with deletions in a soft manner I decided to use a flag instead of the native delete.

mongoose-sugar also attaches the schema definition to the schema itself so that it is easy for me to access that later. This makes it possible for me to serve the description of the database schemas via my REST API.

There is also a specific "refs" utility that makes it easy to make enforced references between models. This is something that takes quite a bit of syntax to do so I decided to add it into this package.

The schema I gave above maps directly to mongoose-sugar. Examine the demo provided with the package to get a better idea of how to do this. Mongoose documentation likely helps as well.

The syntax is somewhat light and quite comfortable to use even. Pretty much the only thing I'm still a bit puzzled about is migrations. I know the situation is a bit different with document databases due to the way they have been designed. Still, this is something I will need to look into.

rest-sugar

As mentioned earlier rest-sugar is an utility that construct a REST API based on a given schema. In order to do this it uses a specific type of query API (with getAll, update etc.) to access the actual models. So far I've defined this only for mongoose-sugar and a memory based storage (a demo). It should be easy to implement that for other drivers too.

The following snippet shows how you might construct a server providing a REST API based on your schemas:

#!/usr/bin/env node
var express = require('express');
var mongoose = require('mongoose');
var sugar = require('mongoose-sugar');
var rest = require('rest-sugar');
var models = require('./models');

main();

function main() {
    mongoose.connect('mongodb://localhost/mongoosedemo');

    var app = express.createServer();

    app.configure(function() {
        app.use(express.methodOverride()); // handles PUT
        app.use(express.bodyParser()); // handles POST
        app.use(app.router);
    });

    rest.init(app, '/api/v1/', {
        'libraries': models.Library,
        'tags': models.Tag
    }, sugar);

    console.log('Surf to localhost:8000');

    app.listen(8000);
}

As you can see the definition is somewhat light. The actual magic lies within the implementation of rest-sugar. If you for some reason wanted to change your schema considerably, you might want to define another init for /api/v2 and use the updated schemas there. Still, it would remain somewhat nice and simple.

Note that you can easily set up some authentication scheme just by passing an extra parameter to rest.init. If you wish to use key based authentication (you should use SSL with this!), pass rest.auth.key('apikey', APIKEY) (where APIKEY is your key) to it. After that the API expects that "apikey" is passed as a part of your query string. It might be interesting to support various other schemes as well. Ideas are definitely welcome!

There are a couple of demos at the repository you might want to examine if this sounded interesting. You will find an example of authentication there too.

rest-sugar-client

So we have got a nice REST API running. We're just missing the frontend bit. This is where rest-sugar-client kicks in. The way it works is quite simple really. All it does is to introspect the metadata provided by the REST API and use that to construct a simple API of its own.

The advantage of this is that we don't need to maintain another set of APIs on the client side. It will definitely stay always in sync. The next step would be to hook this up with some nice frontend library and start coding the user interface.

Even though that sounds simple I find it somewhat cool. In fact I just wrote a simple command line based utility based on this idea. I know curl can do the same thing. This is something more domain specific, though. A web based interface might be interesting to have too.

Conclusion

I hope this post gave you some idea of how I might implement a REST interface and client for it. In my mind introspection is one of the coolest things out there. Just having some metadata around makes it possible to implement nice, generic solutions.

I will likely have to develop a couple of services to see how well this kind of setup works out in practice. There are some aspects I'm a bit doubtful about still (migrations and possible offline usage come to mind). Still, it feels like I managed to develop something that is perhaps useful for some other people too.