October 12, 2014

Shelf Middleware - Adding CORS headers and checking for tokens

When building a web application, it is often necessary to do some sort of authentication / token check on each request and maybe add a CORS header or two to enable cross-site HTTP requests. Usually this is done by some sort of filter or maybe each resource handler takes care of business by itself, using some convenience methods.

When using Shelf such things are handled by Shelf Middleware.

A Shelf Middleware is nothing more than a Shelf Handler that consumes a Handler and return another Handler, meaning you can chain a whole bunch of Middleware together for very complex functionality. In this article I'm going to use the createMiddleware() method to create two Middleware's:

  • One that takes care of CORS, ensuring that all CORS preflight requests are handled and that all responses have the Access-Control-Allow-Origin header set
  • One that checks if a request bears a valid token, and if not, show the user an interface where a token can be obtained

In my previous post I touched on how to use Shelf and how to unit test Shelf request handlers, so take a look at that to learn some more about Shelf, since I'm not going to spend much time on Shelf itself in this post. I'll mainly focus on the Middleware functionality.

Before we move on, it is probably a good idea to clone the shelfmiddleware example code.

Dependencies - pubspec.yaml

Here's what we need:

A quick rundown:

  • args - parse command-line arguments
  • Shelf - Shelf itself
  • Shelf Route - provides means to create routes

Nothing too surprising here, I hope. :)

Where it all begins - bin/shelfmiddleware.dart

All roads lead to main():

We add 3 Middleware to the Pipeline defined at line16, two of which are always added (shelf.logRequests() and addCORSHeaders), and one which we only add if the server is started without the --disableTokens command-line option. If the server is started with the --disableTokens command-line option, then all tokens are considered valid. This is here to demonstrate how easy it is to add Middleware, depending on various circumstances. It is a very flexible system the Shelf people have come up with.

The three Middleware we use are:

  • shelf.logRequests() at line 17 - log all requests and output to STDOUT
  • addCORSHeaders at line 18 - handle CORS preflight requests and add the Access-Control-Allow-Origin header to all responses
  • checkToken() at line 23 - check that requests bear a valid token, and if not, return a page where clients can obtain a token

The final Middleware added to the chain is the last to handle a shelf.Request and the first to handle a shelf.Response, which means that in our example a new shelf.Request is first intercepted by shelf.logRequests(), then addCORSHeaders and finally checkToken. When a shelf.Response is generated, it passes back up the chain, ending with shelf.logRequests(), which incidentally is how shelf.logRequests() is able to tell us how long it took to complete a request to the server - it is the first to see an incoming request and the last to hand off the response.

Lets take a look at how the addCORSHeaders Middleware works.

CORS headers - lib/cors.dart

If you don't know what CORS is, then I suggest you go read this excellent Mozilla Developer Network article on the subject before moving on. Done now? Good.

At line 7 we create the addCORSHeaders Middleware using the shelf.createMiddleware() method. We tell it to hand over all shelf.Request's to the _options() method and all shelf.Response objects to the _cors() method.

All we do in _options() is return an empty shelf.Response with the Access-Control-Allow-Origin header set if this is a preflight request, ie. the request method is OPTIONS, or a null if this is not a preflight request. Returning a null is equivalent to saying "move on, nothing to see here", whereby the shelf.Request is passed on to the next Middleware or Handler in the chain.

In the _cors() method we grab the shelf.Response object and change its headers, always adding the Access-Control-Allow-Origin header.

And this is in its most basic form all you need to CORS enable your server. Naturally you can impose all sorts of restrictions on when and what is CORS enabled, but that is beyond the scope of this article.

Checking for valid tokens - lib/checktoken.dart

Before reading on, please do not consider this example a functional way of handling authentication. It is not. There's a Shelf package for doing authentication, aptly named Shelf Auth. This merely serves as an example on how to use Shelf Middleware, and token checking was what I could come up with.

With that out of the way, lets take a look at our checkToken Middleware:

As with the addCORSHeaders Middleware, we use the shelf.createMiddleware() method to create the checkToken Middleware at line 8, but as you can see this one differs in that we do not set a responseHandler, which of course is because this Middleware only cares about whether or not an incoming request bear a valid token - outgoing responses are of no importance.

The functionality of the _passOrForbid() is mindblowingly simple: We grab the token from the shelf.Request object and return null (meaning: "all OK, pass the request on") if it is valid. If the token isn't valid, we generate a new token at line 17 and then some basic HTML to show to the user, enabling him/her to soldier ahead with a brand new fully functional token.

Worth noting is the response we generate at line 25: shelf.Response.forbidden.

What this does is return a 403 Forbidden alongside the HTML we've created at lines 19 through 23. If you prefer, you could just as well use some sort of redirect here, if that better fits your model. I like the 403 because it sends a clear signal to the user why he/she did not end up at their originally intended destination.

Also observe that this example never respond with a 404 Not Found on requests that contain an invalid token, as all of these requests end up getting caught be the checkToken Middleware. You will only ever get 404's if you request an unknown resource while having a valid token.

The rest of the files - all in one fell swoop

Since this example is all about the two Middleware mentioned above, I'm not going to touch much on the routes, the handlers and the various support methods, but for the sake of completeness I'll post them here anyway:


Two interfaces, both requiring the token path parameter. Nothing else going on here. Each point to their own handler, which we find in:


I think even non-coders can read this. If there's something here that is unclear, please let me know in the comments.


This is where we store valid tokens, and yea this will potentially kill your server if left to its own devices, since I don't do any kind of cleanup on the _validTokens list. Also the tokens themselves are hardly safe, so as I've said before: Don't use this example for ANYTHING serious. It's a security deathtrap!


Simple convenience constants. Note that the Tentacle Response Formatter package can help you determine and set the correct response headers. It's a pretty neat package, that I might write a post about at some point.

Final words

The more I experiment with Shelf, the more I like it. It is clear that the authors of this excellent framework know a thing or two about the world of HTTP. There are still a few rough edges here and there, such as the weird spacing done by the shelf.logRequests() method - I would personally prefer a simple whitespace between each field, since that looks better when the log is piped to syslog, at least on my Debian servers.

Also there seems to be missing a few response types in the shelf.response.dart file, but I guess they'll get added in due time. The most important ones are there, and it is possible to create your own Response objects so it's not a huge issue.

All in all Shelf is awesome.

September 29, 2014

Dart, Shelf and unit testing

I've been using Dart to build server-side applications for a while now. They range from the small and simple to one large beast comprised of many servers and hundreds of REST interfaces. In general its been a pleasant experience, though I do feel that the plain dart.io HTTP server component perhaps is a bit too bare bones. So I set out to find something better, and in my search I found Shelf. It looked to fit the bill. It had some very nice features and it was in the Dart core. I read some more, and I have to admit that I really didn't like it - it seemed overly complicated and, well, even ugly. But I stuck with it, and to nobodies surprise it turned out that I was completely wrong: Shelf is a beauty!

While having fun with Shelf I also decided it was about time I threw myself at the Dart unit test library. I hadn't really used it a whole lot, but since my initial goal was to harden my current crop of Dart HTTP servers, it felt natural to add some Dart unit testing to the mix.

While learning about Shelf, it dawned on me that I might as well put some of the code in a repo and write a short blog about it, so that other beginners can benefit from my experiments. Clone shelftest and read on to learn a bit about how Shelf and Dart unit testing fits together.

Note that this is mostly intended for an audience to whom Shelf and unit testing of a Shelf server is new stuff. Advanced users will probably find little of value here. :)

The goal

This small example aims to deliver a Shelf server with two interfaces and a set of unit tests to go with them. The interfaces are:

GET /user/{userid}
Return the {userid} user as JSON.

PUT /user/{userid}/name
Accept a body in the form {"name":"some name"} and update the {userid} user accordingly. Return the updated user as JSON.

Both interfaces must return a 404 Not Found if the given {userid} path parameter doesn't exist in the "database" and a 400 Bad Request if the request is malformed in some way, ie. the body cannot be decoded to a map or {userid} is not an integer.

Finally I want the unit test output to be in JUnit format, so it plays well with Jenkins.

Dependencies - pubspec.yaml

Lets start by taking a look at the pubspec.yaml file:

A quick rundown is probably in order:

And that is it. As you can see, you don't need a lot to get started with Shelf. I'd suggest that you at least skim-read the documentation for each of the above libraries, so you have a basic understanding of how they work, as I will mainly concern myself with how stuff fits together to create a whole. The nitty-gritty details is for another article.

Where it all begins - main() - bin/shelftest.dart

The main method is wonderfully short and simple:

At line 10 we build a new Shelf.Handler using the Shelf.Pipeline class. A pipeline consists of a series of Shelf.Middleware and a final Shelf.Handler. A Middleware is a method that consumes a Shelf.Request and either passes it on to the next middleware after having done something (as in the case of Shelf.logRequests), or return a Shelf.Response under some circumstances (as in the case of exceptionResponse). Using middleware you can do all sorts of fancy things with requests, before passing them on to either another middleware or a final handler.

Shelf.logRequests is a middleware, and as its name implies it handles logging. It does not conform to the Apache log format, but what it generates is good enough, especially since most Dart daemons will probably live their entire lives behind something like nginx.

exceptionResponse is more interesting, as it enables you to throw a HttpException instead of creating various non successful responses in your handlers. More on this later.

At the bottom we find the routes.handler Shelf.Handler. This method consumes a Shelf.Request and return a Shelf.Response. routes.handler is created in the lib/routes.dart file. Note that you don't have to generate any routes at all - any method that consumes a Shelf.Request can be used as the final handler.

At line 15 we start the server and tell it to listen for requests on localhost:8080. Requests to the server pass through all the middleware and ultimately ends up in the routes.handler where from they are dispatched based on the request method and URL, as defined in lib/routes.dart.

Shelf automatically return 404's for all requests that does not match a route. If a handler throws an unhandled exception Shelf react by returning a 500 response, unless of course it is a HttpException and you're using the exceptionResponse middleware.

Routes - routing requests to handlers - lib/routes.dart

We need to tell our server what to do with incoming requests. In our case we need to define two routes:

Even if you've never worked with Shelf before, it should be rather obvious what is going on here: We define a GET and a PUT route, and using the {userid} syntax we declare that each of these routes must contain a path parameter at the specified location. We can fetch the userid parameter in the handler using the getPathParameter method. In my humble opinion this way to define a route is brilliant, as it reads very well.

The final piece of setting up routes is of course the methods that are supposed to do the actual work, in our case handler.getUser and handler.setUserName. These methods both reside in the handlers.dart file.

Handlers - handling requests and creating responses - lib/handlers.dart

Consuming requests and generating responses is, in general, the task of the handlers we defined in routes.dart - handler.getUser and handler.setUserName. These can be found here:

Starting with getUser we see that all it does is return a user that corresponds with the given {userid}. If it can't find anything (in our db "database"), then it throws a NotFoundException that is then handled by the exceptionResponse middleware we talked about earlier. You don't have to think about creating specific responses for "failure" conditions.

At line 48 you also see a reference to the _getUserId method. This is responsible for extracting the {userid} path parameter and return it as an int. If this fails, it throws BadRequestException which is ultimately handled by exceptionResponse, ensuring us that we get a proper 400 Bad Request returned to the user in case of a malformed {userid} value. If all goes well, we return a Shelf.Response.ok at line 51.

The setUserName method is slightly more complicated, as this one needs to deal with the body of the Shelf.Request. Luckily Shelf makes this easy for us with the readAsString Future. The content of the request is handed over to the _getUserName method, where we try to decode it into a map and then return the value of the name key. If this fails due to a missing or malformed body, then we throw a BadRequestException. If it goes well, then we update the users name at line 70 and return a Shelf.Response.ok with the updated user at line 71.

Take a look at the README.md file for a couple of simple curl commands to try out the server.

A word on lib/user.dart and lib/database.dart

I prefer sharing classes between server and client, but as you can see I'm throwing HttpException's in the User and MapDatabase classes, which would then require that the client also use Shelf Exception Response. This in itself is not a problem, so I decided to not clutter the code further by having one more layer of exceptions. Also worth mentioning is the fact that you can actually use Shelf on the client just fine. See an example here.

Other than that I wont bother talking about the contents of these two files, as they are both quite boring.

Testing it - test/handler_test.dart

When I started writing the tests for this little experiment, I was quite sure it would be somewhat bothersome and boring. It turned out to be dead easy and fun. I've come to enjoy writing Dart unit tests! Who would've known... :D

Here's the file:

First things first: The createRequest method at line 20 is necessary in order to create "fake" Shelf.Request objects to feed into the routes.handler. As you can see, creating a fake Shelf.Request is not exactly rocket science. Whether or not this simple method is enough for all use cases I don't know, but for the tests we need to do here, it's just fine.

The next interesting thing happens at line 27. Yes, this is all it takes to enable JUnit output. Awesome!

The tests themselves are grouped into two groups using the group method. You can have multiple levels of groups, each inheriting the description of the outer group. This is quite handy to avoid repeating the same text over and over in every test.

The tests themselves should be straightforward reading for most Dartisans. There are though a couple of things that are worth noting:

  • The use of the completion method at lines 75 and 139. This is necessary because handler.setUserName return a Future
  • The use of the fail method at line 83 (and other places) to explicitly force a failure of the test. This is nice for those cases where the thing you test must fail if it returns successfully.

If you run the tests (see the README.md file) you should see output similar to this:

Neat eh'? :)

With these few tests in place, we're now guarded against stupid programming errors that make a mess of basic functionality, such as removing a route by accident, suddenly returning 200 OK in places where it should've been a 400 Bad Request and vice versa or not returning 404 Not Found in cases where we have no data to return to the client.

If you keep your handlers small, it should be fairly straightforward to test for most use cases. As handlers grow in complexity, it logically follows that it becomes more and more difficult to test everything, but I still believe that it is so easy to write these tests that there's hardly any valid excuse for not doing it.

Now venture forth and write Shelf servers and unit tests!

I'd like to thank Luiz Mineo for some good advice when I got stuck building the createRequest method. Thanks man! :o)

July 26, 2014

Dart and Google Cloud Datastore

The Google Cloud Datastore is a fully managed NoSQL storage solution that is available to all Google Cloud users. It's a key/value store with a few extra bells and whistles, and I desperately wanted to try it with Dart, so naturally I googled for some examples, and I found this most excellent post called Getting Started With Google Cloud Datastore and Dart that had everything I needed. Brilliant stuff from Adam Singer. I tip my hat to you good sir!

But as you all know, having a functional example really isn't all that fun, so I took Adam's code apart and hacked up my own. Why? Well, do we really need a reason?

So here it is:

Ain't it purdy?

In order for this snippet to do anything for you, you need to be aware of the configuration.dart import and the Config object:

Config is a simple class with a bunch of static members, so I have somewhere to hide all my oauth2 secrets from the world. Note that you will need to grab some stuff from the Google Cloud Developers Console for this example to work:

Config.projectId Available at the GCD Project page
Config.projectNumber Available at the Monitoring/Overview page for your project
Config.serviceEmail Available at the APIS & Auth/Credentials page. Simply create a Service Account using the Create new client ID button
Config.privateKey Generate and download a P12 key from the APIS & Auth/Credentials page. Remember to use the previously created Service Account
Config.scopes Two scopes concatenated and separated by one space: https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/datastore

Special note about the Config.privateKey part: Don't lose the P12 key. This key provides access to your project/app, so guard it carefully. Also you can't use the P12 key, you have to transform it to PEM, so what really hides behind Config.privateKey is actually the contents of the PEM file. You convert the P12 key to PEM with this command:

Again: Guard these files with your life.

With those things set, you should be able to grab the code and run it. The program will connect to your Datastore instance and create a person entity containing the data set in the person map:

If all goes well, the script will return with the ID of this latest entry. You can hit the Storage/Cloud Datastore/Query page at your Google Cloud Developers Console to see the entity. Note that if you feed an entity ID to the script, it will do a simple lookup and return whatever it finds. Again, if it doesn't find anything, a new entity will be created.

Remember that Google Cloud Datastore usage is billable, so don't add a billion entities to the Datastore, unless you're prepared to pay for it.

That's it really. Now all we need to do, is figure out why the whole thing is a bit slow. Hopefully some of the clever guys at cloud@dartlang.org list can help me with that.

On a final note, I'd like to mention that if you run this example from a Google Compute Engine (with proper scopes set at instance creation), then you can simplify the getDatastore() method from this:

to this:

Naturally this also simplifies configuration and completely obliterates the need for the whole P12/PEM key thing. Win!

November 23, 2013

Google Cloud Storage = Lots of Awesomeness

Me and my good AdaHeads buddy Thomas Pedersen gave Google Cloud Storage a whirl the other day, and I can say that we were both very impressed at how easy it was to use. For those of you who don't know what Google Cloud Storage is, here's a quick rundown

Google Cloud Storage is a RESTful online file storage web service for storing and accessing your data on Google's infrastructure. The service combines the performance and scalability of Google's cloud with advanced security and sharing capabilities. It is an Infrastructure as a Service (IaaS), comparable to Amazon S3 online storage service.

Yea, I totally stole that from Wikipedia, but that's besides the point. What is important is that if you haven't tried it, then please go ahead now. It is amazingly good, and it will for sure change how you think about file storage. The accompanying tools and APIs are solid and well documented.

I for one plan on using this for storing a rather massive amount of files I had originally planned on storing on my own servers, but since I really don't feel like managing more hardware, this came as a godsend.

Get started using Google Cloud Storage.

Now go forth and store stuff!

November 21, 2013

Dart console and Google Drive

I wanted to connect to Google Drive from a console Dart application, and after having mucked about with it for a few hours, I finally succeeded. It's not hard, but even so it did take me a while to figure out how to do it. The resulting code looks like this:

I wouldn't mind if some more examples had been made available at the google_drive_v2_api pub.dartlang.org page.

I got my client ID and secret from the Google Cloud Console.

September 21, 2013

Building a simple HTTP server using Dart

So I decided to check out how easy it would be to use Dart to build a HTTP server, and half a cup of tea later, I had this up and running:

It turned out to be amazingly simple. The only "hurdle" I needed to overcome (and that took all of 30 seconds) was to figure out how to dispatch requests - luckily the Route package at pub.dartlang.org did just that.

The code itself should be readable by most, no matter what language you're used to using, and if it isn't, then feel free to ask and I'll do my best to explain.

Dart has once again pleasantly surprised me with its ease of use and straightforward way of doing things. Thumbs up to the Dart developers!

September 15, 2013

Using the Ada Web Server (AWS), part 2

In the Using the Ada Web Server (AWS), part 1 article I showed you how to setup a simple Hello world! server powered by the awesomeness that is the Ada Web Server (AWS) project. In this second part I will show you how to utilize the Templates_Parser module to build your HTML and I'll also give a very short example on how to serve a flat file to visitors.

If you haven't already read part 1, I strongly suggest doing so before proceeding with this article, as we will be building upon the code from part 1 in the following. I will not waste time on how to get the server going, or how to setup a dispatcher. We will jump straight into the template fray.

But first things first:

The instructions on how to compile and run the program are the same as for part 1 of the article.

So what exactly is the point of using templates to generate your content, instead of just building the HTML directly in the Ada code? Well there are several good reasons for using templates:

  • With templates you don't have to re-compile due to changes in the HTML.
  • With templates you get a very strong separation between logic and view.
  • With templates you can easily localize content by simply loading a different template file based on user preferences/IP address/whatever.
  • Your HTML people won't need to learn a single line of Ada. They can stick to what they are good at.

There are probably more good reasons, but the ones mentioned above should be more than enough to convince you that using a template system is a good idea.

Lets take a peek at a very simple template file:

As you can see it looks very much like normal HTML except for the special @_RESOURCE_@ tag (third line from below), which is what turns this otherwise plain looking HTML snippet into a proper template: @_RESOURCE_@ is a template tag that is replaced by some value defined in the Ada code. As you might've guessed this template is used to generate the "404 not found" content, so lets see how the Ada code looks now that we've moved the HTML into a template. This is our new Generate_Content function from the src/not_found.adb file:

If you compare it to the src/not_found.adb file from part 1 you'll notice that the changes really aren't that huge, but lets start at the top: At line 5 we with AWS.Templates; to bring in the templates parser module. With that we now have all the templating power at our fingertips.

The next change happens in the declarative part of Generate_Content where we now have a Use AWS.Templates; line and a Translations : Translate_Set; line. A Translate_Set is basically a dictionary to which you add your tags and their associated values, and to see how that is done we go down two lines to the Insert (Translations, Assoc ("RESOURCE", Resource)); line. Starting from the inside we associate the value of the Resource string with the template tag RESOURCE using the Assoc procedure and then we add the association to Translations with the Insert procedure.

Note that if you insert an association that already exists, then the existing association is overwritten by the new association.

In order to use the Translations dictionary we've created, we must call the AWS.Templates.Parse function:

Only Filename and Translations are required, so that is exactly what we're going to give Parse:

Parse then goes to work, matching all the added associations to the tags found in the template file, which in our case means replacing @_RESOURCE_@ with the value of RESOURCE.

And that is all. The 404 not found content is now fully templated.

Next lets take a peek at Generate_Content from src/hello_world.adb where we use a few more features from the AWS.Templates module:

Let me start by saying that my Fibonacci implementation probably isn't fast nor elegant, but it works and it doesn't clutter the example with a whole bunch of code, and really the interesting part here is not how the Fibonacci sequence is generated but how it is added to the Translations Translate_Set. For this we have the Vector_Tag, which in all its simplicity allows us to build lists of values. As you can see we have two such vector tags: Fibonacci and Position. These two are populated by the Append calls in the for loop:

Append add the values of i and Fibs (i) to Position and Fibonacci respectively. In case you're wondering about what kinds of data you can append to a vector, here's the specification of all the Append procedures:

Note the last one: Yes, you can append a vector tag to a vector tag, making it possible to build multi-dimensional arrays.

Adding our vector tags to Translations is still done using the Insert procedure, so nothing new there.

Finally we generate the content with Parse, where it is worth noting that we've now added the Cached => True parameter. What this does is allow the AWS.Templates module to cache the template itself. If you do this the server no longer read and parse the template file on every hit. The downside is that if you make changes to the template file, you will have to restart the server for the changes to be registered.

Now lets see how we deal with vector tags and do some other tricks in the template file:

Lines starting with @@-- are comments.

At line three we define a macro with the name F. When you call macros you can give them parameters which are then referenced in the macro as @_$n_@, where n corresponds to the N.th. parameter passed to the macro.

Moving on we add the visitors browser to the HTML using the @_BROWSER_@, and just below that we build the Fibonacci table, and here we encounter the @@TABLE@@ tag. Code between the @@TABLE@@ and @@END_TABLE@@ tags are repeated as many times as there are values in the POSITION and FIBONACCI vectors. @@TABLE@@ acts very much like an implicit iterator. Much fun can be had with the @@TABLE@@ tag - this example is the very simplest way to utilize it. Check the manual for more extensive examples.

There's a few more tricks in this template worth mentioning. The templates parser module sports a bunch of constants and filters, one of these being @_NOW_@ which is replaced with a time stamp in the format "YYYY-MM-DD HH:MM:SS". At the next line we reverse the contents of @_NOW_@ using the @_REVERSE:VAR_@ filter. There's a whole bunch of filters available and multiple filters can be applied to tags for pure awesomeness. As an added bonus you can even create your own filters.

And that was all I had about the templates parser module, but before I end this article I'd like to direct your attention to this new dispatcher in src/handlers.adb:

The goal of this dispatcher is to return the contents of the exe/templates/hello_world.tmpl file to the user as text/plain. The Hello_World.Hello_World_Template function takes care of that:

That right there is an Ada 2012 expression function. Since all this function does is call AWS.Response.File we don't really need a body. Expression functions provides a shorthand to declare a function whose body consists of a single return statement. That is IMHO one very nice feature of Ada 2012.

And with that final piece of AWS.Response.File magic I will close this article. Stay tuned for part 3, where I plan on showing you a bit about how to handle HTTP request parameters using the Ada Web Server.