Templating in JavaScript

Templating is one of those basic techniques you'll come by sooner or later when dealing with web development. The following snippet illustrates the basic idea:

var ctx = {
    name: 'jude'
};
var tpl = 'hey {{ name }}! {{ name }} is great';

var res = tpl.replace(/\{\{([a-zA-Z ]*)\}\}/g, function(m, g) {
    return ctx[g.trim()];
});

console.log(res);

As you can see at minimal level we need only two things, template and context. A simple syntax ({{ something }} in this case) is then used to inject the context within the template. That's all.

Since context might not contain just singular values always often templating engines provide a for loop of sort. In addition there may be some logic. Sometimes you may use a real programming language even making the line between a templating engine and a real language a fuzzy one.

Applications of Templating

Stencil by splorp (CC BY-NC-ND)

Templating may be used on both backend and frontend side of the fence. In fact you may use backend templating to render a template which you use in the frontend. So you can get the best of the both worlds should you want to.

Static website generators provide yet another example. Rather than having a dynamic component (ie. some database and associated logic) involved you will just end up with something static to serve.

Regardless of where you apply the idea, the fundamentals still remain quite the same. As I deal mostly with frontend these days I'll illustrate a couple of alternatives over there next.

Templating on Frontend

Frontend of sort by myoldpostcards
(CC BY-NC-ND)

If you notice yourself constructing large HTML structures using JavaScript (ie. $('<div>')) or good ol' document.createElement) you might benefit from converting your code to use templates.

In addition to ending up with less code to maintain you will end up with something that other, less technical people might be able to understand. So it's a win-win.

The simplest way to get started is to define a template as a script (ie. <script id="hey" type="text/template">hey {{ name }}! {{ name }} is great</script>) and then refer to that via your code (ie. $('#hey').text()). As I mentioned you may even generate this template using your favorite backend templating solution.

In case you are using RequireJS, you are in luck. You may refer to templates as dependencies at your modules and include them as a part of your build. There are multiple plugins that make this possible.

Transparency provides an interesting alternative to these. You simply define your template as a part of the regular DOM structure. Transparency binds the data to it. In addition it is possible to define directives to deal with some special logic. How elegant is that?

Conclusion

I hope this brief post gave you some idea how I think about templating. I know it is a very simple concept but it doesn't hurt to spell it out. It's one of those little concepts that makes a big difference when you master it well. Oh, and writing a template engine is a great little exercise. You can bet there are a lot of these around and for a reason!

Middleware Pattern in JavaScript

Tower Complete by teotwawki

If you have ever used Express or some other web framework you are likely familiar with the concept of middleware. It is something that allows us to shape given input before moving into actual logic. Sometimes it may be beneficial to set up a post middleware as this allows you to define triggers based on what has happened. You can see this sort of solution at Mongoose for instance.

A while ago I figured that implementing a middleware scheme would clean up the API of a library of mine, rest-sugar. It is a library that greatly simplifies the creation of RESTful APIs. And I have been polishing it and related technologies lately. Express-style middleware API seemed like a good fit so I implemented that. There are both pre and post filters in place now and I find the interface quite nice.

The following snippet illustrates how the pattern works. Note that it is possible to simplify it further in case you need only one type of filters. In that case you do not need to keep track of context like here.

function init() {
    var handlers = {
        pre: [],
        post: []
    };
    var context;

    // ... your logic goes here

    return {
        use: function(fn) {
            handlers[context].push(fn);
        },
        pre: function(fn) {
            context = 'pre';
            fn();
        },
        post: function(fn) {
            context = 'post';
            fn();
        }
    };
}

// evaluate the middleware at your logic
// Using https://github.com/caolan/async here.
function evaluateHandlers(handlers, req, res, done, data) {
    async.series(handlers.map(function(fn) {
        return function(cb) {
            fn(req, res, cb, data);
        };
    }), done);
}

// sample middleware
function only(method) {
    return function(req, res, next) {
        if(req.method == method) return next();

        unauthorized(res);
    };
}
exports.only = only;

// and usage
var app = express();

...

var api = init(app);

api.pre(function() {
    api.use(only('GET'));
})

api.post(function() {
    api.use(function(req, res, next, data) {
        // trigger now

        // and remember to call next to avoid getting stuck!
        next();
    });
});

Even though the idea is quite simple I like how it gives me a nice way to define extension points to my APIs. You can also define factories to construct middlewares you need. In this case only is such a factory. That is another quite powerful and handy pattern enabled by closures.

Guide - Yes We Jekyll

As I hinted at an earlier post I have been up to something. I composed a guide for Jekyll, the static website generator. Besides Jekyll there is some more general information about web design and the way I approach it. I hope you find it useful! Take a peek at Yes We Jekyll.

Scraping the Web Using Node.js

The Dregs by jazzijava (CC BY-NC-ND)

An important part of a data analyst's work is gathering data. Sometimes you might get it in a nice, machine readable format (XML, JSON, CVS, you name it). There are times when you have to work a little to get the data in a decent format.

Node.js + Cheerio + Request - a Great Combo

As it happens Node.js and associated technologies are a great fit for this purpose. You get to use a familiar query syntax. And there is tooling available. A lot of it.

Disfigured by scabeater
(CC BY-NC-ND)

My absolute favorite used to be Zombie.js. Although designed mainly for testing it works often alright for scraping. node.io is another good alternative. In a certain case I had to use a combination of request, htmlparser and soupselect as zombie just didn't bite there.

These days I like to use combination of cheerio and request. Getting this combo to work on various environments is easier than Zombie. In addition you get to operate on a familiar jQuery syntax so that's a big bonus as well.

Basic Workflow

When it comes to scraping the basic workflow is quite simple. During development it can be useful to stub out functionality and fill it in as you progress. Here is the rough approach I use:

1. Figure out how the data is structured currently
2. Come up with selectors to access it
3. Map the data into some new structure
4. Serialize the data into some machine readable format based on your needs
5. Serve the data through a web interface if you so want

It can be helpful to know how to use Chrome Developer Tools or Firebug effectively. SelectorGadget bookmarklet may come in handy too. If you feel like it, play around with jQuery selectors in your browser. It will be very useful to be able to compose selectors effectively.

Examples

Shady Customer by Petur
(CC BY-NC-ND)

sonaatti-scraper scrapes some restaurant data. It uses node.io, comes with a small CLI tool and makes it possible to serve the data through a web interface.

There is some room for improvement. It would be a good idea not to scrape the data each time a query is performed to the web API for instance. There should be a cache of some sort to avoid unnecessary polling. It is a good starting point, though, given its simplicity.

My other example, jklevents, is based on zombie cheerio. It is a lot more complex as it parses through a whole collection of pages, not just one. It also performs tasks such as geocoding to further improve the quality of data.

In my third example, f500-scraper, I had to use a combination of tools as zombie didn't quite work. The issue had something to do with the way the pages were loaded using JavaScript so the DOM just wasn't ready when I needed to scrape it. Instead I ended up just capturing the page data the good old way and applying some force on it. As it happens it worked.

lte-scraper uses cheerio and request. The implementation is somewhat short and may be worth investigating.

Other Considerations

When scraping, be polite. Sometimes the "targets" of scraping might actually be happy that you are doing some of the work for them. In case of jkl-event-scraper I contacted the right holder of the data and we agreed on an attribution deal. So it is alright to use the data in a commercial way given there is an attribution.

This is just a point I wanted to make as there are times when good things can come out of these sort of things. In the best case you might even earn a client this way.

Conclusion

Node.js is an amazing platform for scraping. The tooling is mature enough and you can use familiar query syntax for instance. It does not get much better than that for me at least. I believe it could be interesting to try to apply fuzzier approaches (think AI) to scraping.

For instance in case of restaurant data this might lead into a more generic scraper you can then apply to many pages containing that type of data. After all there is a certain structure to it although the way it has been structured in DOM will always vary somewhat.

Even the crude methods described here briefly are often quite enough. But you can definitely make scraping a more interesting problem if you want to.

Positive Problems - Embrace Them

The Lightner by pfv. (CC BY-NC)

There is an interesting trend I have noticed when working with startups and small clients. Not all problems are created equal. There are those you absolutely must resolve. And then there are those that would be nice to have resolved but that presume the whole apparatus works as expected and depend on those others.

And this exactly is what brings me to the concept of positive problems. If there is one thing where some startups go wrong, it's premature scaling. They start to think about scaling the technical side and architecture before they have proven the whole concept even works in practice. I consider scaling related issues a great example of positive problems. You should be glad to have those.

Before you have been able to prove the basic mechanics of your business model there isn't much sense in prematurely optimizing your system. It simply doesn't make sense to scale a zero. It remains a zero still. Rather optimize learning and find the winning combination, the right value proposal for the right market. Even achieving this can be a huge leap for a small company.

In short when prioritizing what to develop and when optimize learning in the beginning. It can be useful to have those scaling issues somewhere in back of your mind but I would not worry about it before you can find something that works. You can switch gears then. I know there will be people that disagree with this statement. In the end it comes down to what can you afford.

jquery.tokko.js - TOC and Anchors for Your Documents

Frustration based development is so fun. jquery.tocco.js is one example of that. I simply wanted a nice way to generate a hierarchical table of contents and some anchors for a lengthy guide of mine.

I will discuss that guide later on a separate post but you can take a peek at it if you are feeling curious. In the meantime you can play around with tokko. There are some cat images even so it can't be that bad. Enjoy!

Thoughts on Instanssi 2013

It's a mancave, alright

This was the second year in row in which I visited a local demoscene event, Instanssi. As usual it was a nice little event. Around hundred people, mostly male geeks, showed up and it was a great fun. I even gave a brief talk about that Elovalo web IDE thing.

On retrospect I should have developed more functionality for the conference but due to work pressures I just couldn't pull it off this time. We did manage to get our web interface for the cube to work so it is possible to control it remotely. This yields some very interesting opportunities for further development.

The highlights of the events for me include a talk on functional reactive programming using Elm (slides in Finnish), Setok's talk on startups and a meeting with Andrew Paterson of Pixelache. Pixelache acts as an aggregator of sort and provides some cross-pollination between various maker communities and such we have in Finland and even in neighboring countries (Russia and Baltic area countries in particular).

Frontseats were often empty, a Finnish tradition(?)

Overall the event was well organized. There were some issues with networking but nothing too major from my point of view. I agree with Setok's opinion that perhaps there should be some kind of a socializing space next time.

It's embarrassing for us Finns to go bug other people when they seem to be so busy with their computers. In a way that's part of the culture, though. Do keep in mind that Instanssi is a non-alcoholic event so we don't have that social lubricant around (not that I personally use any).

I hope the event will be arranged again next year. I know it's hard work to organize these sort of events but in the end it's worth it. It's a nice highlight that kind of signifies the upcoming Spring for me. It's not that far off anymore.

I recommend checking out the demo entries. Laser Berger's 1984 is a fun listen in particular. Nice retro feeling there. Kind of reminds me of Miami Vice for some reason.