Quandy Factory Newsfeed - Blog

Services-First: A Better Way to Build a Web Application (DRAFT)

2012-02-02T12:00:00Z

Introduction

The best way to find yourself is to lose yourself in the service of others.

-- Mahatma Gandhi

When you're building a web application, it's a powerful design heuristic to build a REST web service API, and build your application on top of your service.

The functionality your application needs to perform should all be accessible or at least exposable as web services. This enforces a responsible design approach that will pay real dividends in maintainability, extendability and reduced technical debt.

It may also pay unexpected dividends in generating value over and above the principal goal of your web application. By thinking about your application design as a web service, you open up the possibility that the real value you deliver is not what you thought your service was going to be, but some subsidiary or extraneous solution you develop that turns out to be a) useful to others and b scalable/profitable.

The Infamous Yegge Platform Rant

Last October, Google engineer Steve Yegge posted a long, thoughtful rant on his public Google+ page in which he argued that Google "does everything right" - except platforms, which he argues Google still doesn't really understand. Rather, Google provides products, not services.

Yegge's argument was directed internally at his fellow Googlers, and he took the essay down after realizing that he had accidentally posted it publicly. Luckily for us, copies are still available with Yegge's blessing.)

Yegge contrasted his former employer, Amazon, which overwhelmingly adopted a services-first approach after founder and CEO Jeff Bezos ordered every department in the company to start exposing its functions to the rest of the company as a web service. According to Yegge:

[T]he first big thing Bezos realized is that the infrastructure they'd built for selling and shipping books and sundry could be transformed an excellent repurposable computing platform. So now they have the Amazon Elastic Compute Cloud, and the Amazon Elastic MapReduce, and the Amazon Relational Database Service, and a whole passel' o' other services browsable at aws.amazon.com.

According to Yegge, this services-first approach has allowed Amazon to serve as a platform on which other companies build their businesses.

Dogfooding

Operating as a platform drastically increases the accessibility of a company's products and increases that likelihood that someone will build a "killer app" on the platform.

In Amazon's case, the killer app is Amazon's own business, which runs on the AWS platform (though a number of third party companies have also built their applications on top of the AWS platform).

Yegge stresses the vital importance of building on your own platform:

The Golden Rule of Platforms, "Eat Your Own Dogfood", can be rephrased as "Start with a Platform, and Then Use it for Everything."

Yegge points out that it might be painful to maintain the discipline to eat your own dogfood, but it is far more painful and expensive to turn a monolithic application into a platform after the fact.

If you delay it, it'll be ten times as much work as just doing it correctly up front. You can't cheat. You can't have secret back doors for internal apps to get special priority access, not for ANY reason. You need to solve the hard problems up front.

According to Yegge, Amazon went through the painful exercise of transforming itself from a product company into a platform company because of necessity:

[I]t took an out-of-band force to make Bezos understand the need for a platform. That force was their evaporating margins; he was cornered and had to think of a way out. But all he had was a bunch of engineers and all these computers... if only they could be monetized somehow... you can see how he arrived at AWS, in hindsight.

Amazon Web Services

Here is how Bezos himself explained the strategy in his latest annual letter to shareholders:

Our technologies are almost exclusively implemented as services: bits of logic that encapsulate the data they operate on and provide hardened interfaces as the only way to access their functionality. This approach reduces side effects and allows services to evolve at their own pace without impacting the other components of the overall system. Service-oriented architecture - or SOA - is the fundamental building abstraction for Amazon technologies.

Amazon launched its first web service - Elastic Compute Cloud (EC2) - in 2006. Since then, it has followed up with a long list of additional services:

CloudFront - low latency global CDN
DevPay - accounts receivable service
DynamoDB and SimpleDB - schemaless nosql key/value store
ElastiCache - scalable in-memory cache
Elastic MapReduce - distributed mapreduce processing for huge datasets
Flexible Payment Service (FPS) - digital payment service
Mechanical Turk - on-demand human intelligence market
Relational Database Service (RDS) - relational database
Simple Email Service (SES) - bulk and transactional email service
Simple Notification Service (SNS) - send notifications
Simple Storage Service (S3) - data storage and retrieval
Route 53 - Domain Name System service

None of these service are central to Amazon's core business, which is selling books and other consumer products directly to consumers. However, they all began life internally as essential peripheral or supporting functions.

Today, the company can sell these web services publicly because Bezos had the foresight to insist that the company conduct its internal affairs via web services.

The company is secretive about how much revenue it earns directly from its suite of web services, but the best estimates are that revenue reached $500 million in 2010 and $750 million in 2011. Web service revenue is expected to reach $1 billion in 2012 and $2.6 billion in 2015.

Services-First

No matter what you build, you will develop expertise in a number of related, peripheral functions that support your primary product. If you develop your product in a services-first fashion, you allow for the possibility to expose those peripheral functions to third parties.

No initial product idea survives first contact with the market. Every successful business is successful because it was able to adjust - sometimes wildly - its product and business strategy in response to real-world feedback. Startup gurus call this "pivoting", and a founder's ability to pull it off is decisive in the survival of the business.

Pivoting doesn't mean throwing away what you've built and starting over. Rather, it means re-purposing what you've built to reach a more promising market.

Many successful startup products started out as components of larger, less focused products. In his book The Lean Startup, entrepreneur Eric Ries calls this the "zoom-in pivot". Other businesses started out assuming they would be selling to one market and ended up attracting interest from a different sector altogether.

A services-first approach decouples your product from the platform it's built on, allowing you more flexibility to change it as required. It also allows you to provide the platform itself to clients as an additional source of value.

Designing Your Web Application

So when building your web application, it's a powerful design heuristic to build a web service first, and then build the application on top of it.

Why a Heuristic?

Application design is non-deterministic. There is not one "right" way to do things. Ten different designers given the same problem will create ten different solutions - and any number of them can be 'good enough'.

When you can't reason your way to a proof, you have to fall back on a less certain, more exploratory approach. A heuristic is a method of approaching the problem that tends to help you produce good solutions but falls short of being a solution itself.

Think of a heuristic as a general rule of thumb rather than a specific prescription.

Design Goals

For the end user, a web service should be:

Useful - The most elegant web service in the world won't interest anyone if it doesn't solve real problems.
Usable - Similarly, the most useful web service won't be adopted if people can't figure out how to use it.
Explorable - The easier it is to navigate and discover what a service does and how to use it, the more it will be adopted.
Accessible - In the case of a web service, accessibility means a variety of clients can use and interact with it.
Combinable - The purpose of web APIs is so clients can combine data and functionality from a variety of sources to build new things.
Flexible - Your own product is not the only thing that can be built with your platform. Your API should be open enough to allow for use cases you couldn't imagine.

For the web service developer, a web service should offer and/or encourage:

Clean, consistent organization of data and functionality
Separation of different layers in the application stack
Flexibility to change the application built on it
Clarity of abstractions

Finally, the developer is also an end user and will also benefit from usefulness, usability, discoverability and so on.

Design Considerations

With these goals in mind, I propose the following design considerations. Again, since design is non-deterministic, these considerations can help you to make decisions in the open-ended design process that will tend to point you toward a better final product.

Design for Adoption

The entire purpose of an API is for people to use it. If you're not sure how to design something, use this question as a guide: What will make it easier for users to understand how this works and how to use it?

If you force yourself to be a user, you will be more inclined to look at the API from the user's perspective.

Design for Maintainability

Developers spend more time fixing, refactoring and modifying existing code than they spend creating new code. Your API should be designed in such a way that it is easy to dive in and work with existing code. One way to achieve this is through what the Rails developers call convention over configuration. A architecture that builds on established standards will be easier to maintain.

Representational State Transfer (REST)

REST is a style of web service architecture based around clients and servers that closely models the way HTTP works. In a RESTful system, a client makes a request for a resource on a server, and the server issues a response that includes a representation of the resource.

The concept was formalized in 2000 by Roy Fielding, one of the architects of Hypertext Transfer Protocol (HTTP), in his doctoral dissertation. It is not surprising, then, that REST and HTTP mesh very smoothly.

A RESTful client-server system is stateless, meaning each request against the server contains all the information the server needs to process it; and cacheable, in that the server can specify whether and for how long resource representations can be cached either locally on the client or on intermediate servers between the client and the server.

HTTP

Hypertext Transfer Protocol (HTTP) is a stateless protocol based on a client requesting a resource across a network and the server providing a response. As such, an HTTP transaction entails a request and a response. The request goes from the client to the server, and the response goes from the server back to the client.

HTTP Requests and Responses

In HTTP, the client makes an HTTP request to the server and the server issues an HTTP response.

Requests

An HTTP request has three parts:

The request line, which includes the HTTP method, the URL and the HTTP version: GET /users/1 HTTP/1.1
One or more optional HTTP headers, which are key/value pairs with metadata about the data being requested and/or provided.
An optional message body, which is data being sent from the client to the server as part of the request. E.g. in a POST request, the message body will include the data that the server should use to create a new resource.

Response

An HTTP response also has three parts:

The HTTP Status Code, which indicates the status of the requested resource: HTTP/1.1 200 OK
One or more optional HTTP headers, which are key/value pairs with metadata about the response being provided.
An optional message body, which is a representation of the resource that was requested (hence the name "Representational State Transfer").

Idempotence

This funny-looking word is really important: a request is idempotent if making it multiple times has the same effect as making it just once. Read that again if you have to.

A request is not idempotent if issuing it more than once has a different effect than issuing it once. For example, a POST request to add a comment to a document is not idempotent: issuing the POST request twice adds the comment twice, so that the document contains two comments with unique URLs.

URLs and Resources

In a RESTful architecture, each URL represents a resource. This is vitally important: a resource is a noun, an object, and not the action performed on it.

A RESTful web service has multiple endpoints - one for each resource. (Contrast SOAP, which has only one endpoint and puts everything else - objects, methods, parameters, etc. - into the XML payload.)

If you find yourself creating URLs like /create_user, you're doing it wrong. Instead, create a URL like /users and map your user object to that URL.

Remember: a resource is a noun, not a verb.

HTTP Methods

If a resource is a noun, the HTTP Method is the verb. What you do to the resource with your request depends on what method you use. If a URL is an object, the HTTP method is the action you execute on that object.

HTTP Methods
Method	Action
GET	Retrieve a resource
POST	Create a new resource
PUT	Update an existing resource
DELETE	Delete an existing resource

In our user example, execute an HTTP POST request on /users to create a new user. The server should respond with the specific URL of the user you created: /users/1.

To view a representation of that resource, issue an HTTP GET request on the user's URL: /users/1.

To update the user, issue an HTTP POST request on the user's URL with the new user data in the request body.

To delete the user, issue an HTTP DELETE request on the user's URL.

GET Method

To retrieve a resource, issue an HTTP GET request. GET requests are idempotent (see below), which means making a GET request multiple times does not cause any change in the resource that is requested.

GET requests do not include a message body, but GET responses usually do.

POST Method

To submit data to be processed, issue an HTTP POST request. POST requests require a message body, i.e. the data to be processed.

For example, if there is a resource called /articles and you want to add a new article, issue a POST request to /articles with the content. The server should create a new subsidiary URI under /articles - for example, /articles/1 - and assign that URL to the content you sent with your POST request.

Important note: POST requests are not idempotent, meaning multiple POST requests will create multiple resources with unique identifiers.

PUT Method

To update the resource at an existing URL, issue an HTTP PUT request on that URL. For example, if there is a URL /articles/1 and you want to replace the content served at that URL, issue a PUT request to that URL with the new content.

Important note: PUT requests are idempotent, so issuing 2 or 5 or 50 identical PUT requests will have the same effect on the resource as issuing just one PUT request. Like POST requests, PUT requests include a message body (the resource to be placed at the URL).

DELETE Method

To remove a resource (and remove its accompanying URL), issue an HTTP DELETE request. DELETE requests should be idempotent, i.e. issuing 1 or 2 or 5 or 50 identical DELETE requests will delete exactly one resource. DELETE requests do not require a message body.

HTTP Headers

Both HTTP requests and responses can include optional HTTP Headers, or key/value pairs that supply meta-data about the request and the response.

Accept Header

An HTTP request can include an Accept header that specifies what media types the client will accept in a response.

In a REST web service, the request should include an Accept header with the preferred media type - e.g. application/json - and the server should attempt to fulfill the client's preference in its response, given its capabilities.

If you are willing or required support multiple formats (e.g. JSON, XML, YAML), the best way for the client to specify what format they prefer is via the Accept header.

Here are some examples of Accept headers:

JSON: Accept: application/json
XML: Accept: application/xml
YAML: Accept: application/x-yaml

Note: YAML has an x- prefix because it is not a formalized MIME type. For thoroughness, you could accept all four possibilities:

application/x-yaml, application/yaml, text/x-yaml, text/yaml

HTTP Status Codes

HTTP has a great way of telling the client whether the request was successful or an error has occurred: HTTP status codes. Every HTTP response includes a status header with an HTTP status code that indicates the status of the request.

This tells the client whether the request was successful or not, and what to expect by way of a response. It's important to send the right status code.

401 Unauthorized (Image Source: Flickr)

I also include HTTP status codes and error descriptions in the response body. A REST purist might call this overkill, but remember that you want to make your API as self-documenting and easy to use as possible.

For the purposes of your web service, there are three categories of HTTP status codes you need to worry about: success codes, client error codes, and server error codes.

Success Codes

HTTP success status codes provide useful information about successful requests:

200 OK - A GET request was successful at retrieving a resource.
201 Created - A POST request was successful at creating a resource.
202 Accepted - This means the request was accepted but the response is not ready. Useful for queued or otherwise deferred processes.

Client Error Codes

400 Bad Request - The request data could not be parsed properly, or a value was missing or invalid.
401 Unauthorized - The user requested an access-restricted resource but authentication either failed or was not provided.
404 Not Found - The user tried to request a resource that does not exist.
405 Method Not Allowed - User tried to execute an HTTP method on a resource that does not allow it. E.g. a POST request against /users/charlie instead of /users.
406 Not Acceptable - The request came with an Accept header for a media type that the server cannot provide in the response.
410 Gone - A previously available resource has been removed permanently.
429 Too Many Requests - Useful if you've rate-limited your API for a given user.

Server Error Codes

500 Internal Server Error - Try to avoid this one. It generally means your web service is broken.
501 Not Implemented - User has tried to send a request method that your server doesn't recognize.
503 Service Unavailable - This tells the client the service disruption is temporary.

Designing a RESTful Web Service

Don't Reinvent the Wheel

HTTP was invented to allow clients to request resources from servers across the internet and for those servers to respond with representations of those resources. The entire internet runs on it. It's flexible, powerful and offers a clean abstraction model based on resources and methods.

Don't re-implement what HTTP does in an ad-hoc way on top of HTTP. Just use HTTP.

Many web services only use HTTP as a network tunneling protocol, and then pass objects, methods and parameters inside an 'envelope' that contains all the information. This type of web service architecture includes SOAP and other Remote Procedure Call (RPC) formats.

Web services that actually use HTTP the way it was designed are called Representative State Transfer (REST) web services. Unfortunately, REST is widely misunderstood and a lot of web services that call themselves "RESTful" aren't.

Discoverable API

In a RESTful web service, a GET request on a base API URL returns a list of resources available in the API. Critically, each resource has a direct URL. This makes it very easy for developers to step into your API and figure out what is available.

Now, you may be thinking you've seen this pattern before. You're right: it's exactly how every single website works. Go to the homepage, and what do you find? Links to the other resources on the website!

A RESTful approach makes your URLs hackable. If your API user is looking at a URL like: /cats/miss_mew, they should be able to lop off the /miss_mew and do a GET request against /cats to return the base collection of cats (each with its own URL, of course).

REST Resource/Method Matrix

At a conceptual level, a RESTful web service API is a matrix of resources and methods that exposes the functionality of the service to third party applications. Below is an example of what that matrix might look like.

Again, I will repeat that actions are not mapped to URLs. A resource is an object - a noun - and the action inheres to the HTTP method - a verb - not to the URL.

As a result, the same resource URL can serve different responses (corresponding with different actions) depending on the HTTP method used.

REST Resource/Method Matrix
Request				Response			Idempotent
Resource	URL	Method	Request Data	Server Action	Response Body	Status Code	Idempotent
Users	/users	GET		retrieves list of users	list of users and associated URLs	200 OK	Yes
Users	/users	POST	user details	creates a new user	new URL and user representation	201 Created	No
User 9001	/users/9001	GET		retrieves details for user 9001	user representation	200 OK	Yes
User 9001	/users/9001	PUT	new user details	updates user details	updated user representation	200 OK	Yes
User 9001	/users/9001	DELETE		deletes user 9001	returns status	200 OK	Yes
User 9001	/users/9001	GET		checks that user 9001 does not exist	Not found error message	404 Not Found	Yes

Use JSON

JavaScript Object Notation (JSON) is a data serialization format that delivers an optimal combination of the following criteria:

Lightweight - less boilerplate means less data transfered across the network
Readable - Humans can easily see and understand the content of a JSON object
Flexible - allows various nestable data types: object (dictionary), array (ordered list), string, integer, float, boolean
Interoperable - every programming language and framework under the sun can generate and consume JSON.
Convertable - JSON can be converted into a native object more easily than XML.

Here's an example of a response to an API base URL GET request, represented in JSON:

{
    "ok": true,
    "status_code": 200,
    "resources": [
        {
            "url": "/authors",
            "resource": "authors"
        },
        {
            "url": "/articles",
            "resource": "articles"
        },
        {
            "url": "/blogs",
            "resource": "blogs"
        },
        {
            "url": "/comments",
            "resource": "comments"
        },
        {
            "url": "/events",
            "resource": "events"
        }
    ]
}

Unless you've got a compelling business or technical reason for using a different media type, stick with JSON.

Search

Since grammatically, "search" can be a noun as well as a verb, it's okay to have a URL called /search. Arguably, the best way to filter the response a server delivers to a GET request on /search is to use a querystring:

/search?doctype=article&keyword=hello%20world

It is possible to do this without a querystring - something like:

/search/doctype/article/keyword/hello%20world

But that's silly and pedantic - and arguably wrong, anyway. It's harder for your users to guess and it's harder for you to parse.

Pagination

When your user does a GET request on /api/cats, do they really want a list of 7 million cats? Do you really want to have to serve that and support the processing and bandwidth involved?

Again, I don't think there's anything wrong with using querystrings here:

/api/cats?offset=101&limit=50

Similarly, I don't think there's anything wrong with using terminology that is familiar to developers who work with databases.

Versioning

Your API is a contract you have made with your users. If you plan to introduce a change to your API that breaks backwards compatibility, you need to do so in such a way that it doesn't break existing clients.

There is no easy, obvious way to do this that will make everyone happy, satisfy REST and satisfy the goal of making your API usable and discoverable.

There are a few different ways you can version your API, all of which have pros and cons.

Versioning in URL

Many popular web APIs do this: /api/v1/cats. If you roll out a version 2 of your API that would break clients using version 1, you expose it at api/v2/cats.

Pro: it's easy for users to understand and doesn't break existing clients.
Con: results in multiple URLs for the same resource; locks versioning into the URL; multiplies API maintenance issues.

Versioning in Querystring

Some APIs do this: api/cats?v=1.

Pro: default URL doesn't need to include version (defaults to newest version); one URL for each resource.
Con: clients that don't explicitly specify a version will break when the API is updated.

Versioning in Accept header

REST purists recommend putting the version in a custom Accept header: Accept: application/vnd.company.app-v1+json

Pro: default URL doesn't need to include version; one URL for each resource; changes do not break clients; version is a formatting issue so it's the "right" place.
Con: more esoteric and difficult for developers to discover how it works.

Use Secure HTTP

I don't want to say too much about security (though I do recommend the REST Security Cheat Sheet - Draft document from the Open Web Application Security Project (OWASP) as a handy reference), but I will say a quick word in support of Secure HTTP - or HTTPS - rather than plaintext HTTP.

HTTPS is a bit slower, but on a web service, you're not making multiple requests to pull down javascripts, CSS, image files and so on, so you can afford to take a small hit for an encrypted connection.

The payback in client security is worth it.

Summary of Benefits

It's extremely easy to lose yourself in theory and get sucked into the purity movement (remember XHTML?). If REST doesn't deliver practical results, it doesn't matter how elegant, pure or canonical it claims to be.

In short, a RESTful API is discoverable and navigable. With no background knowledge, a client can do a GET request on the root URL to get a list of resources with URLS. It's easy to dive right in and quickly figure out what is available.

Better URLs

Stable, persistent URLs allow for resource bookmarking and better search indexing.

Sane, well-named URLs are more descriptive than arcane or arbitrarily named URLs.

Hackable URLs allow for better resource discovery.

Developers Understand HTTP

REST is essentially just HTTP, and we already understand HTTP (with an important caveat, below).

The API developer doesn't need to waste time and energy (badly) reinventing/re-implementing what HTTP does on top of HTTP.

The client doesn't need to waste time and energy learning a bunch of ad hoc RPC protocols.

Our Tools Understand HTTP

Just about every programming language can natively perform an HTTP request and parse the response (with an important caveat, below).

Problems with REST

REST comes with its own issues, of course, and it would be disingenuous to gloss over them.

Requires Proper Understanding of HTTP

Much as I wrote earlier that REST is just HTTP and we all understand HTTP, the honest fact is that we don't understand HTTP all that well.

It's true we've all been using it for 20 years now, but for various historical, cultural and educational reasons, there's a constant tension between understanding and using our protocols the way they were designed and knowing just enough to build something that gets the job done.

That latter imperative lowers the barrier to entry into web development, but it also leads to the same avoidable mistakes being made over and over again in applications that require a more professional approach: SQL injection, XSS, hacked cleartext passwords, and so on.

We've all heard horror stories about web applications that didn't understand HTTP and blew up when a search engine bot innocently crawled a GET request to /delete_user.php?username=Ryan and thereby wiped out the entire database.

For a newbie developer who doesn't really understand HTTP very well, a web service with URLs like /api/create_user is more immediately obvious than a web service with URLs like /api/users that require a POST request to create a user.

However, this is changing as the benefits of REST become more widely known and more developers start to take advantage of its defining characteristics.

One thing is for certain: once you've developed and built on a REST web service, you'll finally understand HTTP inside and out.

Tooling

SOAP advocates argue that mature tooling makes it easy for a service provider to deploy a WSDL and for clients to consume it and execute web service methods as if they were local function calls.

Of course, the reality is less rosy: leaky abstractions, incompatible data types, lack of interoperability between, say, a Java WSDL and a C# SOAP client, and so on.

In far too many cases, I've had to hand-roll an ad-hoc SOAP client and build my own XML templates when a supposedly push-button WSDL turned out to be platform-dependent. Instead of saving time by consuming a WSDL, I was stuck painstakingly reverse-engineering the RPC formatting details that the interface tried unsuccessfully to hide.

Still, SOAP has that enterprise-friendly turnkey appeal, even if the delivery doesn't fulfill the promise.

REST, on the other hand, is sorely lacking in push-button tooling - on both the framework development and client side.

Even tools that ought to know better tend to abstract away the HTTP methods from the developer and generically treat every request as a GET request - unless it comes with form data or hits a kludgy /process_form kind of an URL.

We've come a long way from the cgi-bin form processing scripts of the '90s, but we still have a way to go. The good news is that this changing as developers come to recognize the benefits that come from understanding and using HTTP the way it was designed.

Frameworks

Most web application frameworks weren't developed with HTTP in mind, but rather with the minimal subset of HTTP that that is comprised of viewing web pages and filling out forms. Data transfer and processing, even in web applications designed for human users, tends unconsciously to follow an RPC model.

There are already a few that have embraced the full functionality of HTTP more deeply.

I'm not that familiar with Ruby, but I understand Rails 3 is designed to work RESTfully right out of the box with its routing DSL.

Similarly, the lightweight Sinatra framework understands resources and methods natively and simply:

require 'sinatra'

get '/users' do
  # show users
end

get '/users/1337' do
  # show a user
end

post '/users' do
  # create a user
end

put '/users/1337' do
  # update a user
end

delete '/users/1337' do
  # delete a user
end

In Python, Django is not RESTful by design has some good REST plugins.

The lightweight Python framework I use, web.py, is REST-friendly at its core: resources map to classes and HTTP methods map to class methods.

import web

app = web.application(urls, globals())

urls = ('/users/(.*)', 'User')

class User(object):
    def GET(self, name):
        # get a resource

    def POST(self, name):
        # create a resource

    def PUT(self, name):
        # update a resource

    def DELETE(self, name):
        # delete a resource

In PHP, you've got Zend_REST, Recess, Tonic and others that understand and support RESTful API design.

The .Net framework has Windows Communications Foundation (WCF). Java has Play RESTlet, Jersey and so on.

Client Tools

Nearly every language understands HTTP, but too many default HTTP modules understand HTTP in a crippled, semi-literate way that reflects how most browsers use HTTP: fetch pages or fill out forms.

In Python, there are no less than three HTTP libraries that ship with the language: urllib, urllib2 and httplib - and none of them make the full expressiveness of HTTP particularly accessible.

Fortunately, third party developers have stepped in with libraries like httplib2 and requests, which do a much better job of exposing all the HTTP methods in a sane, consistent manner.

Similarly, REST-aware clients are appearing in other languages as well.

If you're a .net shop, you've got tools like RestSharp.

Of course, JavaScript understands HTTP natively, perhaps better than some other, more "mature" and "full-featured" languages - possibly because JavaScript lives in browsers and breathes HTTP.

var xmlhttp=new XMLHttpRequest();

// GET request
xmlhttp.open("GET", "/users", true);
xmlhttp.send();
xmlhttp.onreadystatechange = function() {
    if (xmlhttp.readyState == 4) {
        alert('GET request successful'); 
        // response is in xmlhttp.responseText
    }
}

// POST request
var jsonObject = {"username": "ryan", "fullName": "Ryan McGreal"};
var jsonString = JSON.stringify(jsonObject);
xmlhttp.open("POST", "/users", true);
xmlhttp.setRequestHeader("Content-Type","application/json");
xmlhttp.send(jsonString);
xmlhttp.onreadystatechange = function() {
    if (xmlhttp.readyState == 4) {
        alert('POST request successful'); 
        // response is in xmlhttp.responseText
    }
}

// PUT request
var jsonObject = {"username": "ryan", "fullName": "Ryan G. McGreal"};
var jsonString = JSON.stringify(jsonObject);
xmlhttp.open("PUT", "/users/ryan", true);
xmlhttp.setRequestHeader("Content-Type","application/json");
xmlhttp.send(jsonString);
xmlhttp.onreadystatechange = function() {
    if (xmlhttp.readyState == 4) {
        alert('PUT request successful'); 
        // response is in xmlhttp.responseText
    }
}

// DELETE request
xmlhttp.open("DELETE", "/users/ryan", true);
xmlhttp.send();
xmlhttp.onreadystatechange = function() {
    if (xmlhttp.readyState == 4) {
        alert('DELETE request successful'); 
        // response is in xmlhttp.responseText
    }
}

JavaScript also has a panoply of frameworks and libraries that make HTTP requests even easier:

// yay jQuery!

$.get("/users", function(response){ alert('GET request successful'); });

$.post("/users", data, function(response){ alert('POST request successful'); });

$.put("/users/ryan", data, function(response){ alert('PUT request successful'); });

$.delete("/users/ryan", function(response){ alert('PUT request successful'); });

So the tools for creating and consuming REST web services are steadily improving, while the tools for creating and consuming SOAP web services are over-engineered and IMHO over-rated.

In the meantime, REST makes up in discoverability and navigability what it lacks in automated tooling.

Browsers

Of all our tools for working with HTTP, the most universal is the browser. It's the easiest way to browse a web service API, particularly if it works like the web itself. After all, the URL is just a resource on a server accessible via HTTP, and browsers are specifically designed to make HTTP requests and present the responses to users.

However, like other established HTTP-based tools, most browsers are only good at GET and POST requests, not PUT or DELETE requests. Even then, POST requests generally require an HTML form that the user can fill in and submit.

In addition, most browsers are not good at rendering representations in formats other than HTML or XML.

I use Firefox as my main browser (yes, I'm old-fashioned), and I've discovered a few add-ons that make exploring web APIs a lot easier:

HttpFox - an HTTP analyzer.
JSONView - pretty-prints JSON response objects in the browser window.
Modify Headers - Add, modify and filter the HTTP request headers sent to a web server.
RESTClient - Visit and test REST/WebDav services.

REST is a Spectrum

Ultimately, it makes sense to talk about an API being more or less RESTful and you probably shouldn't kill yourself to reach 100.00000%.

Like anything else, REST purity is subject to diminishing returns. If you get the important stuff correct - URLs are resources, methods are actions, representations include URLs - you'll get to enjoy the main benefits of a REST design approach without tearing your hair out.

For each choice, ask yourself whether the benefit of going more RESTful justifies the cost. Keep in mind that your goal is to make your API as usable and discoverable as possible.

References

Classification of HTTP-based APIs

http://nordsc.com/ext/classification_of_http_based_apis.html
Richardson Maturity Model - A model (developed by Leonard Richardson) that breaks down the principal elements of a REST approach into three steps. These introduce resources, http verbs, and hypermedia controls.

http://martinfowler.com/articles/richardsonMaturityModel.html
REST APIs must be hypertext-driven - Roy Fielding's attempt to get people to understand what he was talking about when he defined REST.

http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
RESTful API Design - Pragmatic REST - Webinar by Apigee.

https://www.youtube.com/apigee#p/c/5/R8SIxZVaai4
REST Security Cheat Sheet - Draft document from the Open Web Application Security Project (OWASP)
RESTful Web Services: The Basics - IBM developerWorks article

https://www.ibm.com/developerworks/webservices/library/ws-restful/
General Principles for Good URI Design - StackOverflow answer

http://stackoverflow.com/a/1619677/682547

Python String Formatting With Dictionaries

2012-01-04T12:00:00Z

My mind is duly blown. I never realized that the traditional printf-style string formatting in Python - the kind that uses the % operator - supports the use of a dictionary as well as a tuple.

The following works:

>>> data = {
    'title': 'Python String Formatting With Dictionaries',
    'author': 'Ryan McGreal',
    'summary': 'In Python, you can use dictionaries instead of tuples to populate values via classic string formatting.',
    'content': '<p>My mind is duly blown...',
    'author_bio': 'Ryan McGreal lives in Hamilton with his family and works as a programmer and writer.',
}

>>> template = """<!DOCTYPE html>
<html lang="en">
<head>
<title>%(title)s</title>
</head>
<body>
<article>
<header>
<hgroup>
<h1>%(title)s</h1>
<h2>By %(author)s</h2>
<h3>%(summary)s</h3>
<hgroup>
</header>
%(content)s
<footer>
%(author_bio)s
</footer>
</article>
</body>
</html>"""

>>> print template % data

As an ultra-lightweight templating engine, this is pretty sweet: it's fast, simple, supports Unicode, and has no third-party dependencies.

Designing a RESTful Web Application

2011-09-30T12:00:00Z

Introduction

I'm working on a couple of projects that involve building a web service, and I decided early on that because of our business constraints - having to communicate with a variety of different systems of varying levels of sophistication - it made sense to keep the web service as simple and accessible as possible.

That pointed me toward designing a RESTful web service that transmits data in a simple format over straight HTTP. After all, just about any programming language imaginable can make an HTTP request. I also decided to go with JSON for the data format, in part because I've been experimenting lately with CouchDB and appreciate both the simplicity and flexibility of JSON and the fact that you can find a JSON parser for any language.

This blog entry is my attempt to get all the concepts of RESTful web service design straight. There's a good chance that some of this information is wrong; and if you notice something, please let me know about it. I'll investigate your argument and update the essay as applicable.

With that in mind, here we go.

Representational State Transfer

Representational State Transfer, or REST, is a model for designing networked software systems based around clients and servers. In a RESTful system, a client makes a request for a resource on a server, and the server issues a response that includes a representation of the resource.

The concept was formalized in 2000 by Roy Fielding, one of the architects of Hypertext Transfer Protocol (HTTP), about more which below, in his doctoral dissertation. It is not surprising, then, that REST and HTTP mesh very smoothly.

Hypertext Transfer Protocol

Hypertext Transfer Protocol (HTTP) is a stateless protocol based on a client requesting a resource across a network and the server providing a response. As such, an HTTP transaction entails a request and a response. The request goes from the client to the server, and the response goes from the server back to the client.

HTTP Requests

An HTTP request has three parts:

The request line, which includes the HTTP method (or "verb"), the uniform resource identifier (URI), and the HTTP version. E.g.

GET /article/1/ HTTP/1.1
One or more optional HTTP headers, which are key/value pairs that characterize the data being requested and/or provided.
An optional message body, which is data being sent from the client to the server as part of the request.

HTTP Responses

An HTTP response also has three parts:

The HTTP status code, indicating the status of the requested URI, e.g.

HTTP/1.1 200 OK
One or more optional HTTP headers, which are key/value pairs that characterize the data being provided.
An optional message body, which is the data being returned to the client in response to the request.

Resources

HTTP deals in resources. Each URI points to a resource on the server. Think of URIs as nouns, not verbs, with one URI for each resource.

For example, if you want to add the ability to create an article, it might be tempting to create a URI called /create_article. This is wrong, because it conflates the object (the resource) and the action (creation). Instead, it makes more sense to have a resource called /article and a method that lets you create articles.

HTTP Methods

HTTP defines several methods, or "verbs", to execute on a resource: HEAD, GET, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT, and PATCH. However, the following four are most commonly used in web services:

GET Method

To retrieve a resource, issue an HTTP GET request. GET requests are idempotent (see below), which means making a GET request multiple times does not cause any change in the resource that is requested.

GET requests do not include a message body, but GET responses usually do.

POST Method

To submit data to be processed, issue an HTTP POST request. POST requests require a message body, i.e. the data to be processed.

For example, if there is a resource called /article and you want to add a new article, issue a POST request to /article with the content. The server will create a new subsidiary URI under /article - for example, /article/9001 - and assign that URI to the content you sent with your POST request.

Important note: POST requests are not idempotent (see below), meaning multiple POST requests will create multiple resources with unique identifiers.

PUT Method

To place content at an existing resource, issue an HTTP PUT request. For example, if there is a URI /article/9001 and you want to replace the content served at that URI, issue a PUT request to that URI with the new content.

Important note: PUT requests are idempotent, i.e. issuing 1 or 5 or 50 identical PUT requests will have the same effect on the resource. PUT requests must include a message body (the resource to be placed at the URL).

DELETE Method

To remove a resource (and remove its accompanying URI), issue an HTTP DELETE request. DELETE requests should be idempotent, i.e. issuing 1 or 5 or 50 identical DELETE requests will delete exactly one resource. DELETE requests do not require a message body.

Idempotence

This funny-looking word is crucial to designing an effective web service. A request is idempotent if issuing it more than once does not change the resource state beyond issuing it just once. Read that again if you have to.

For example, a DELETE request is idempotent if the first request deletes a resource at a URI, and the second request does nothing because the resource at that URI is already deleted.

For another example, a PUT request is idempotent if the first request updates a resource at a URI, and the second request updates the same resource in the same way at the same URI.

A request is not idempotent if issuing it more than once does change the resource. For example, a POST request to add a comment to a document is not idempotent, if issuing the POST request twice adds the comment twice (so that the document contains two identical comments with separate URLs).

PUT vs. POST

My original understanding of HTTP methods was that you would use PUT to create a resource and POST to update it. This seems in keeping with common sense, but it breaks down when you apply the all-important filter of idempotence.

An interesting discussion on Stack Overflow tackles this issue, but what convinced me to change my mental model was this observation:

POST creates a child resource, so POST to /items creates a resources that lives under the /items resource. Eg. /items/1.

PUT is for creating or updating something with a known URL.

This collision between the inclination to regard PUT as creating and POST as updating is a significant source of confusion about how to well-design a RESTful system, and deserves more attention.

Furthermore, it's tempting to assume that the HTTP verbs line up precisely with the SQL CRUD verbs, but while they're superficially similar, they're not identical. Treating them as such leads to this kind of gotcha.

It's important to keep the logic of HTTP methods separate from the logic of SQL queries, and to develop specialized appropriate logic between the two domains that ensures the data processing on the server produces responses that satisfy the requirements of the HTTP methods (particularly in respect to idempotence).

Conceiving the Web Service: A Resource/Method Table

Again, note well that actions are not mapped to URIs. A resource is an object, a noun, and the action inheres to the HTTP Verb, not to the URI. As a result, the same resource URI can serve different responses (corresponding with different actions) depending on the HTTP Verb.

REST Resource/Action Matrix
Request				Server Action	Response	Idempotent
Resource	Parameters	Method	Data	Server Action	Response	Idempotent
/article		POST	article details	creates a new article	returns confirmation and id	No
/article	/id	GET		gets article details	returns article details	Yes
/article	/id	PUT	new article details	updates article details	returns confirmation and updated article details	Yes
/article	/id	DELETE		deletes an article	returns confirmation of deleted article	Yes

This is the RESTful way to organize a web service: URIs are objects and HTTP Verbs are actions performed on those objects.

HTTP Response Data Formats

Every web server is also a RESTful web service, accepting GET and POST requests to particular resources, and then performing actions and serving data in response.

A conventional web server delivers its data in HTML format (text/html), with related Javascript (text/jss), CSS (text/css) and image files. HTML is an excellent format for marking up textual data for human use, but it has very limited expressive power for structuring data beyond simple documents.

The most common formats used to transmit structured data across HTTP are XML and JSON, with an honourable mention for YAML.

XML

XML, or eXtensible Markup Language, is a markup (i.e. tag) based syntax based on SGML for formatting structured, text-based data. XML is a format in which to create domain specific markup languages that define particular data structures.

For example, RSS and Atom are XML language standards defined to structure documents published to websites so that the documents can be 'syndicated' to feed readers and third party sites for display.

Likewise, the default underlying structure of Microsoft Office documents since Office 2007 is an XML language called OOXML.

XML structures data by defining elements, properties, data types and allowable nesting rules in an XML schema called a Document Type Definition, or DTD.

A given XML document specifies which DTD schema should define its structure with a Document Type Declaration, or DOCTYPE.

A given XML document can reference multiple DTDs by using namespaces.

Here is a sample XML file containing contact information about a person, taken from Wikipedia.

<?xml version="1.0" encoding="UTF-8"?>

<Person>
    <firstName>John</firstName>
    <lastName>Smith</lastName>
    <age>25</age>
    <address>
        <streetAddress>21 2nd Street</streetAddress>
        <city>New York</city>
        <state>NY</state>
        <postalCode>10021</postalCode>
    </address>
    <phoneNumber type="home">212 555-1234</phoneNumber>
    <phoneNumber type="fax">646 555-4567</phoneNumber>
</Person>

XML must be:

Well-formed - elements are properly nested, tags are properly closed and match case, special characters are escaped, and Unicode characters are encoded; and
Valid - its elements and attributes match the rules defined in the DTD.

An XML Language called XSLT can be used to map XML documents into other markup languages, e.g. HTML.

XML is in wide use in applications that transfer structured data over the internet.

JSON

JSON, or "JavaScript Object Notation", is a lightweight data format introduced in 2001 by Douglas Crockford.

Pronounced "Jason", JSON is based on JavaScript object literal notation, a syntax for creating objects in JavaScript by literally describing their properties and methods. Here is the JSON equivalent to the XML code in the previous section.

{
    "firstName": "John",
    "lastName": "Smith",
    "age": 25,
    "address": {
        "streetAddress": "21 2nd Street",
        "city": "New York",
        "state": "NY",
        "postalCode": "10021"
    },
    "phoneNumber": [
        { "type": "home", "number": "212 555-1234" },
        { "type": "fax", "number": "646 555-4567" }
    ]
 }

While the XML above contained 367 characters (not including indentation), the equivalent JSON contains only 272 characters - only three-quarters as large.

JSON supports lists (ordered sets of values) and dictionaries (unordered collections of key/value pairs) with arbitrary nesting and various data types: number, string, boolean, list, *object and null.

Like XML, JSON is also in wide use in applications that transfer structured data over the internet.

A major advantage over XML is that the syntax is much simpler and less verbose, which makes it lighter across networks as well as more human-readable.

Mature JSON parsers are available for a wide range of programming languages in addition to JavaScript. Python, for example, includes a json parser as part of its standard library (as of version 2.6; earlier versions can use the third-party simplejson library).

A decent JSON parser converts an object back and forth between the programming language's native data types and their JSON equivalents. That way, an application can receive a JSON object, convert it into a native object, process it natively, and then convert the final result back to JSON to be dispatched elsewhere.

YAML

YAML, pronounced to rhyme with "camel", is a recursive acronym meaning "YAML Ain't Markup Language". Whereas JSON is a more minimal data format than XML, YAML takes minimalism to an extreme, eschewing quotation marks, brackets and curly braces altogether in favour of significant indentation and line breaks.

The YAML equivalent to the XML and JSON contact examples above would be:

firstName: John
lastName: Smith
age: 25
address: 
    streetAddress: 21 2nd Street
    city: New York
    state: NY
    postalCode: 10021
phoneNumber: 
    - type: home
      number: 212 555-1234
    - type: fax
      number: 646 555-4567

That works out to just 239 characters - including the significant white space.

REST Best Practices

A RESTful web API ought to be discoverable by its users. One crucial way to do that is by making sure that each resource in your API includes URLs to subsidiary URLs.

Here is an example, using JSON:

{
"articles": 
    [
        {
            "title": "My First Baguette",
            "description": "After a month of reading about how to make baguettes, I finally took the plunge today.",
            "date_published": "2011-07-11",
            "url": "http://quandyfactory.com/blog/81/my_first_baguette"
        },
        {
            "title": "Tenn. Passes Controversial Lawnmower Theft Bill",
            "description": "The lawnmowing industry has successfully lobbied the Tennessee State Government to pass a groundbreaking law making it a criminal offence to loan your lawnmower to a neighbour.",
            "date_published": "2011-06-02",
            "url": "http://quandyfactory.com/blog/78/tenn_passes_controversial_lawnmower_theft_bill"
        }
    ]
}

When you do this, you make it easy for API users to discover your API structure and functionality without having to keep referring to obscure documentation.

Special thanks to Adrian Duyzer for reading a draft of this essay and setting me straight on the respective roles of PUT and POST.

My First Baguette

2011-07-11T12:00:00Z

I got up at 5:00 AM this morning to make baguette. First of all, I made a poolish, which is a pre-ferment made from flour, yeast and water. I made this batch and left it on the printer in the basement to rise slowly over the day.

Fresh poolish set to ferment on our basement printer

I came home after work to a mature bowl of poolish that had the consistency of the sticky goo that kids get in an egg-shaped plastic container and which invariably ends up getting ground into the carpet, staining the cushions and stuck to the cat's hair.

The poolish after 12 hours of fermenting

Close-up of the fermented poolish

I mixed up a batch of dough using the poolish, more flour, water, yeast and salt, and kneaded it for around 15 minutes. I coated the dough in oil, put it in a covered bowl and left it to rise for an hour or so.

Once it had risen, I split it into four pieces, pre-shaped them into ovals, wrapped them in cellophane and left them for another twenty minutes.

Then I unwrapped them, rolled them out into baguettes, and arranged them in a makeshift couche, or "bed", which is supposed to be a heavy linen cloth that keeps the loaves in shape as they undergo their second fermentation.

My makeshift couche, a red hand towel that turned out to be rather on the sticky side

I ran into trouble getting the loaves out of the couche and onto the pan, as they tended to stick to the material, even though I had floured it beforehand. As a result, one of the loaves ended up looking rather like that bulbous Orc Lieutenant from the Return of the King movie.

In any case, I cranked the oven to 450 degrees and added a pan of water for steam. As the oven heated up, I wondered what the hell I was thinking baking in the middle of July - "hell" being the operative word. (Note to self: next time, hold off on learning to bake a new kind of bread until September.)

I baked the loaves for around 25 minutes. When they came out, the one looked like an orc but the best looking one looked almost like an actual baguette.

Four cooked baguettes, at least one of which actually looks like a baguette

Finally, the baguettes cooled and I was ready for my first taste test. I served myself a slice with brie de meaux and some red pepper confit. It did not taste quite right, but is certainly better than I expected.

The taste test: fresh baguette with brie and red pepper confit

More experiments to follow in the near future!

Tenn. Passes Controversial Lawnmower Theft Bill

2011-06-02T12:00:00Z

From Yahoo! News:

The lawnmowing industry has successfully lobbied the Tennessee State Government to pass a groundbreaking law making it a criminal offence to loan your lawnmower to a neighbour.

The practice of lawnmowersharing is claimed, according to an industry-funded study, to cost the industry $10 billion a year in lost revenue.

"Now that everyone who wants to enjoy a mowed lawn has to come clean and buy or rent their own lawnmower, we can finally put an end to the harmful piracy that has been driving the lawnmowing industry to the brink of collapse," said Dr. Lawrence Angelo, an industry spokesperson.

The barbecue industry is watching this closely as it attempts to secure passage of a law that would uphold barbecue terms-of-use restrictions preventing barbecue owners from flagrantly cooking food for dinner guests without a multi-user licence.

How to Write a Blog Post

2010-07-26T12:00:00Z

The most basic guideline to writing a blog is this: Provide value for your readers. Valuable writing is personal, informative, instructional, revelatory, entertaining, and engaging. The object of your blog is not to promote your organization but to serve the interests of your readers.

Don't think of it as an alternative channel for press releases or other marketing activities; people can see a sales pitch coming a mile away. Instead, think of it as a way to build relationships with readers by sharing your expertise in a way that helps them and encourages them to share your articles more widely and come back for more.

Topics

There's no predicting which topics will generate the most interest, but variety of topics and approaches will give your blog the broadest set of opportunities to engage readers.

If you stick with it, blogging becomes a habit that integrates itself into your workflow and actually helps you by engaging readers to discuss, expand, scrutinize and clarify your own thoughts.

Relax. Think of blogging as a conversation, not an assignment. A blog entry is closer in tone to a personal correspondence than a term paper. Think about subjects, issues or events that you would enjoy discussing with your peers.
Write for yourself. Write about things that interest or challenge you. The single most-read article I've ever written is a technical guide on designing web applications, which I wrote to clarify the underlying principles so I would understand them more fully.
Try to surprise. Most nonfiction writing is goal-oriented and persuasive, but the term "essay" comes from the French verb essayer, which means "to try". An exploratory essay is a formalized attempt to understand an issue rather than defend a conclusion. Exploratory essays often lead in surprising directions, which makes for a compelling read.
Solve a problem. Your day is filled with problems and challenges that are similar to those faced by other people. Share some of the innovative ways you have solved particular problems and invite readers to share theirs. Another popular blog entry I've written was a summary of lessons I've learned trying to deal with comment trolls.
Find and explore a niche. The internet is a big enough medium that even seemingly narrow subjects, if well-developed, can attract lively reader communities. This also provides opportunities to transfer ideas and practices from one niche to another.
Explain a statistic. Modern citizens are bombarded with statistics delivered out of the blue and out of context. You can humanize statistics by explaining what they mean in concrete terms and by sharing anecdotes that sketch the human faces behind the numbers.
Share your experience. Your value to your organization goes beyond a narrow job description and encompasses all the experiences, values and connections you bring with you to work. Tap into those so your writing reflects your broader competence.
Entertain. Good blog writing is like good conversation: refreshing, invigorating, and entertaining as well as informative. Feel free to have a bit of fun.
Revisit older posts. Over time, with new information and greater experience, our opinions often change and evolve. It's worth going back to older posts periodically to reassess them - what is still true, and what has changed. Obviously you won't be able to do this right away.

Writing and Tone

Write in first person. You are writing as a person as well as on behalf of an organization. Allow your writing to reflect your personality and your personal voice. Don't be afraid to refer to yourself as "I" if you're writing about something you've done.
Use the active voice. Your blog is written by people, about people and for people. Don't drop the people out of your writing by relegating them to the tail-end of your sentences. Figure out who is the active agent in a sentence and move that person or entity forward into the subject.

Passive: "Ideas were gathered from community residents on how to use the vacant land."
Active: "Community residents shared their ideas on how to use the vacant land."
Have an opinion. Readers know you are a human and expect you to have opinions. Good opinions are reasonable and defensible but encourage discussion.
Add value. Don't just re-post a link to something else. At the very least, add your own commentary that expands, clarifies, corrects or otherwise enhances the linked content.
Vary the length of your posts. Sometimes a short piece with a pithy observation or bit of commentary can stand on its own. Other issues deserve a more in-depth treatment. Both are welcome and keep the blog interesting.
Leave some room. Your article doesn't need to be exhaustive or anticipate every possible critique. A well-written article covers an important aspect of an issue but leaves room for readers to add to the conversation by expanding, clarifying, and challenging what you've written.

Editing

Always check spelling and grammar. Typos and other flagrant errors look sloppy. Aim for the sweet spot that feels like conversational English - the terrain that lies between the strictures of formal writing and the morass of split infinitives and grocers' apostrophes.
Edit for clarity and brevity. Stephen King recently shared the simplest advice he ever received for revising: Draft 2 equals draft 1 minus 10 percent. He also advises, "Kill your darlings" - by which he means sentences that serve only to display their own cleverness.
Remove noise words. We all habitually pepper our writing with such tics as "to be sure", "in order to", "bottom line", "liaise", "going forward", "in the loop", "two cents", "for what it's worth", "in my [humble] opinion", "ducks in a row", "outside the box", "at the end of the day", "vis-a-vis", and a profusion of gratuitous modifiers (e.g. "very unique") that communicate nothing but hyperbole. Get rid of them.
Define Jargon. Some business-speak is a necessary evil. If you have to use industry-specific terminology, write the terms and acronyms out in full and define them for readers.

Presentation

Write a Punchy Title. Clear, descriptive, humorous titles catch the reader's attention and help search engines determine the article's content. Warning: don't go overboard. A hyperbolic title that oversells the content will leave readers feeling betrayed.
Keep paragraphs short. Big walls of text are hard to scan on a computer screen. Keep paragraphs short - between one and three sentences - and discrete in terms of their contents.
Include subtitles. Subtitles within an article, particularly a longer one, break up the text into manageable chunks.
Use appropriate layout techniques. If you want to draw attention to a key word or phrase, make the text bold. If you have a list of items, use lists (like the lists in this guide). Use bullet-point lists for your items unless they need to be in a specific order, in which case use numbered lists.
Summarize. After writing your article, write a one- or two-sentence summary and post the summary at the top, just beneath the title. Combined with a punchy title, a clear summary will help readers decide whether your article is worth reading.

Ubuntu 10.04 First Thoughts

2010-05-01T12:00:00Z

Last night I upgraded my Acer Aspire One from Ubuntu 9.10 to 10.04 (I use the regular edition, not the netbook edition). 10.04 is a Long-Term Support (LTS) release, meaning Canonical promises to support the desktop edition for three years and the server edition (which comes without a GUI layer) for five years.

It took all night to download the upgrade (watching the time remaining on the download status dialog brought me back to the halcyon days of the Windows 95 file-transfer time remaining status) but the upgrade itself was painless.

So far, here's what I think:

Boot Time

This is the most immediate, obvious improvement in 10.04. True to their commitment, the Ubuntu developers have made big strides in shaving as much as they can off startup and shutdown times.

Startup is visibly faster than 9.10, which didn't really improve much on 9.04. From the power button through the kernel selection to the login screen runs a cool 35-40 seconds.

Add another six or seven seconds after logging in to bring up the desktop, and another two or three seconds to auto-connect to the local wifi network.

One interesting note: 8.10 and 9.04 displayed long lists of command-line messages during bootup, while 9.10 eliminated that for a splash screen. Now I get command-line messages again, albeit only about eight lines and for only a second or two.

Shutdown

Shutdown is even nicer: six seconds flat.

I have only two small gripes with the shutdown process:

The indicator applet on the panel splits the user menu and shutdown menu into two separate dropdowns, but they still sit right beside each other as if they were a single item. It will take a little getting used to.
The default behaviour for shutdown is to pop up a confirmation dialog, but like 9.10 you can't right-click on it to access preferences and turn off the dialog. The easiest way to fix this is to open up a terminal and start gconf-editor. (If you don't have it installed, fire up the Ubuntu Software Centre and install Configuration Editor.) Expand "Apps" in the tree menu on the left, select the "indicator-session" setting on the right side and check "suppress_logout_restart_shutdown" to turn off shutdown confirmations.

Appearance

It's not brown. Really, that's the main thing. Beyond that, if you've upgraded after already customizing the appearance of 9.10, most of your settings remain intact (with one notable exception, below).

I tried out the new default Ambiance theme and didn't like it. It's just a little too purple and dreary for me. But as before, you can customize the hell out of your window styles via System -> Preferences -> Appearance.

Window Controls

In fairness, the window titlebar button positions are less bad than they were when Canonical first proposed moving them to the left side of the window chrome. In the first iteration, the buttons were in the order [-][_][X], but in the release version they're in the order [X][_][-] with the close button on the very left.

Yet I just don't see a benefit to moving them to the left side that justifies the aggravation of having to unlearn a habit that goes back nearly 20 years. After all, I originally chose Ubuntu in part because it seemed to be the easiest transition from a background in Windows.

Luckily there's an easy way to fix it. Open a terminal and run gconf-editor. In the tree menu on the left click on Apps -> metacity -> general. Double-click on the "button_layout" setting on the right side and replace the text with: menu:minimize,maximize,close to restore the previous behaviour.

A word of warning: if you go into System -> Preferences -> Appearance and change the theme, the window buttons will reset to the left side again.

Wifi

Ubuntu 8.10 and 9.04 were a bit painful to get wifi working on an Aspire One. The only thing that worked for me was to download madwifi and compile from source. Every time a major system update installed, I had to reinstall madwifi (checkinstall didn't work for me).

This changed with 9.10, in which wifi Just Worked, and it still works in 10.04. In fact, it seems to connect considerably faster now - and the little connection icon in the NetworkManager applet looks prettier than the previous one. (With the icon in 9.10, I always had to double-check whether I was looking at the wifi or battery status.)

Ubuntu One

I decided to try out Ubuntu One, Canonical's cloud storage offering. Roughly equivalent to significant players (e.g. Dropbox), Ubuntu One offers 2 GB of cloud storage for free with the option to buy 50 GB for $10 per month. Also like Dropbox, you can share individual files and folders publicly.

Note: you access Ubuntu One through your Me menu, not through System -> Administration or System -> Preferences.

Setting up an account is easy enough, though the web interface felt slow and unstable. Actual syncing seems pretty smooth, though I haven't yet had a chance to try and modify files using different systems at the same time. More to come as I play with this.

The syncing options seem fairly impressive. You can even install a Firefox addon to sync your bookmarks.

Ubuntu One also comes with a music store provided by 7Digital. The selection is passable if nowhere near exhaustive, and the prices are not bad. I'd rather see songs in the $0.25-$0.50 range - for me, that's the break-even point against the opportunity cost of downloading free music - but we're moving in the right direction.

The music store also provides MP3s, which will grate Free Software purists but serves the rest of us just fine.

The store integrates closely with the Rhythmbox music player, which suffers most of the aggravations of a full-featured music player but at least can boast that it's not nearly as bloated or clumsy to use as iTunes.

For iPod users who are sick of iTunes and/or itching to run Linux instead of Mac or Windows, you can actually use Rhythmbox to load and sync your iPod.

Firefox

There was a hullabaloo when Canonical announced that they were switching the default search engine on Ubuntu Firefox from Google to Yahoo, but they switched back before the final release. Hey, money talks when you're giving your product away for free.

As for Firefox itself, Ubuntu 10.04 ships with version 3.6.3. The lag between Firefox releases and the version available through Ubuntu's package manager was a fairly long-standing aggravation, which led to workarounds like Ubuntuzilla to get newer versions working.

I hope this is a sign that Canonical is committed to staying on top of new Firefox releases.

Problems

I only had two real software failures.

Brasero flat-out stopped working, returning an "unknown error" on any attempt to burn a CD or DVD. It looks like a lot of people are having this problem. Sooner than try to fight my way to a fix, I just started using Gnomebaker instead.

Gtk-Gnutella also stopped working. The version in the Lucid repository is two points behind the current version and throws an "ancient version detected" alert; it also fails to connect to the filesharing network.

My attempts to compile the newest version from source (I tried both checkinstall and a straight compilation) produced an application that would crash seconds after opening. I finally gave up and switched to Frostwire.

Otherwise, everything continued to work as expected.

Summary

There's certainly a lot more to explore before I've gotten to the bottom of this new edition, but so far I'm impressed. It's fast, solid and stable to use, the ratio of Just Works to Needs Hacking is approaching 1:0, the upgrade retained most of my previous settings without breaking anything, and several gripes I previously had with Ubuntu have been resolved.

Aside from a few annoyances that are fairly easy to work around, I have no major complaints.

Shared Awareness: A Better Way to Manage Comment Trolls

2010-04-15T12:00:00Z

Trolling

As everyone who has spent more than five minutes on the internet knows, in the absence of personal accountability, some people inevitably turn into assholes.

John Gabriel's Greater Internet Dickwad Theory

Not many, mind you, but even one determined troll on an internet forum can be enough to bring the entire affair crashing down.

The term "troll" comes originally from the fishing method of dangling a shiny lure out the back of a boat while putting along just quickly enough to grab the interest of a fish. The fish that can't resist the lure is snared on the hook and eaten.

(Of course, the term also connotes ugly, hairy brutes who lurk under bridges waiting to snatch unwary passers-by.)

Trolls posting anonymously on message boards absolutely love to provoke outrage. The troll's objective is not to change anyone's mind; rather, it's to write in such a way as to elicit a sequence of increasingly angry, frustrated replies that drag the discussion further and further off-topic until the original thread is in shambles.

Crude trolls subsist on the meager attention given to their vulgarity; but sophisticated trolls can keep a debate going for days by dancing on the fine line between feigned reasonableness and deliberate obtusity. They are by far the more damaging to online discourse.

What makes trolls disruptive is not the trollish comments themselves, but the chain of outraged replies they manage to elicit.

Duty Calls

Successfully managing trolls requires understanding why this happens and breaking the chain of replies.

Shared Awareness

Trolls attract replies by exploiting the general absence of shared awareness on most mediated communications platforms. Shared awareness is a special state of meta-knowledge among collaborators that allows them to act cooperatively on the knowledge.

If I know something and you know something but I don't know that you know it and you don't know that I know it, we can't build on that knowledge even though we both know the same thing.

Shared awareness is not reached until: I know something, you know something, I know that you know, you know that I know, I know that you know that I know, and you know that I know that you know. (Whew!)

Once we all know that we all know the thing, we can then act on it. Without that shared awareness, our ability to put the information to use is sharply curtailed.

Now, sometimes the absence of shared awareness is a blessing. There are many potentially socially awkward situations - say, someone in a room full of people passes wind - in which the absence of shared awareness of who passed wind means everyone can take a position of plausible deniability.

Not only does the culprit avoid embarrassment, but also the other parties don't have to deal with the discomfort of having to address the incident. After all, most people don't like confrontations.

Someone is Wrong on the Internet!

In a conventional online comment system, several participants may each know something, but they do not necessarily have shared awareness about it (everyone knows that everyone knows).

When a troll posts a comment, for example, other participants may decide independently that the comment is inappropriate and/or wrong and/or offensive, but they have no way of knowing whether anyone else feels the same way.

A conscientious forum-goer could just ignore the troll and hope for the best, but silence is affirmation, and a comment left to stand unchallenged starts to look like it represents the prevailing view of the forum participants.

Hence, the conscientious forum-goer feels no choice but to reply to the troll, if only to assure others that at least someone else disagrees with it.

Now the fish is hooked, and a skillful troll can keep that fish on the line for a long time.

Collateral Damage

In the meantime, other fish get scared away by the thrashing.

Pointless, unending debates between trolls and earnest opponents do two things:

They disrupt and crowd out legitimate discussion; and
They deter other potential commenters from getting involved, lest they be trolled as well.

Eventually, the community starts to dissipate altogether. At this point, the troll - really, a parasite (if you'll forgive the switched metaphor) - abandons the dying host and sets off in search of a fresh victim.

The important thing to remember is that the damage takes place not only because the troll posts a comment but also because an earnest commenter decides to reply.

In turn, earnest commenters get into it with trolls because in the absence of shared awareness, they have no other way to know whether the community as a whole accepts or rejects the troll's arguments.

Worst of all, the very dialogue with the troll, undertaken for the purpose of invalidating the troll's position, actually deters others from sharing their own positions and clearing up the matter.

Three Points of Attack

The three necessary conditions for a troll disrupting a forum are:

An absence of shared awareness;
A troll; and
A well-meaning opponent to debate with the troll.

It's safe to generalize that any online forum will have both trolls and well-meaning opponents willing to debate with them. Since most online forums also don't have shared awareness, that means all three necessary conditions are likely to be present and disruption is likely to follow.

Since all three conditions are necessary for disruption to take place, we have three points of attack in trying to prevent it:

Establish shared awareness;
Block the trolls; or
Block the well-meaning replies.

Of the three options, #1 is the most promising. If you can establish a shared awareness that the troll is inappropriate - i.e. everyone knows that everyone knows that the troll is not representative - then well-meaning opponents no longer feel obliged to reply.

In turn, since the objective of a troll is to provoke replies, a troll who can't get anyone to take the bait will eventually give up and move on to another forum.

So how do we go about establishing shared awareness on a text-based communications forum?

Clear Understanding of Rules

The first step to shared awareness is for everyone on the forum to have a clear understanding of the rules and a clear set of expectations on now to behave and how others should behave. It's not enough just to say of trolling that we know it when we see it

The rules should be simple, straightforward, fair, and agreeable to the members. Ideally, they should participate in determining what the rules will be. That way, the members have a sense of investment and ownership in the policies that govern their actions on the forum.

However, the rules should probably include some variation on the following:

Maintain a civil and polite tone, even when disagreeing with someone.
Argue from evidence and cite your sources.
Do not use rude or insulting language.
Do not post comments that are needlessly inflammatory, seek to provoke an emotional reaction from others, are attempts to disrupt and derail the discussion, or abuse evidence and reasoning to defend an unjustifiable conclusion.
Do not post comments that are off-topic.

The important thing is that the participants feel invested in community standards that define them as civil, reasonable and responsible. Again, most people - even on the internet - like to think of themselves in these terms and actually value the personal exchanges they get to have on socially functional online forums.

A pseudonymous screen name might be less personal than a real name, but it still inheres to a real personal identity, and people can still form relationships while communicating on the internet - if their community is successful in establishing and maintaining a shared awareness of their values.

Communicate Without Posting a Reply

By themselves, codes of conduct are only useful insofar as people see them being applied fairly and consistently.

For inappropriate comments, there needs to be a mechanism for members of the community to flag them as inappropriate without actually posting replies; and that flag must be visible to everyone so that an individual looking at the comment can see how many other people have already flagged it.

If you have used reddit, Digg, or Hacker News, you may have already have experienced such a mechanism at work. In these three link sharing sites, registered users can up-vote or down-vote both submitted articles and user comments.

While each site works a little bit differently, a given article's or comment's score reflects some combination of the number of upvotes, the number of downvotes, the 'reputations' of the users who assigned upvotes and downvotes, and the amount of time that has passed since the article or comment was posted.

Some sites introduce an additional constraint in that registered users do not gain the ability to downvote comments until after they have first attained a minimum threshold of reputation based on how other users have upvoted and downvoted their articles and comments.

Hacker News also starts fading the text colour of comments with net negative scores. The benefit to this is that it sends a strong visual message about how the community feels about the comment.

Remove the Incentive

So what happens after you implement this form of community moderation? Earnest, well-meaning commenters can see for themselves that the community has rejected a troll and hence no longer feel obliged to set the record straight in a reply.

Eventually, the troll gets bored at not being able to elicit any more replies and moves on.

This is the key to successful community moderation: establishing shared awareness removes the incentive to reply to trolls, which in turn removes the incentive to troll.

To borrow a phrase from John Gilmore, the other forum members learn to treat the trolls as damage and route around them. Eventually the trolls give up.

It might take a while to establish shared awareness, but the forum admins must be patient and remain steadfast in the face of what is likely to be frenetic opposition from the trolls before they finally quit.

Have no doubt: the trolls will immediately understand what is happening, and will fight to undermine it with every rhetorical tool at their disposal, accusing you of censorship, tyranny, megalomania and anything else they can think of. They will fling a litany of abuse at you in the hope that some of it will stick.

Free and Voluntary

The best way to fend off these attacks is to ensure that the moderation is free, voluntary, and non-censorious. The following guidelines seem to work well:

Regularly reinforce community standards by having the policy displayed prominently on the site and reminding users of their important role in maintaining those standards.
Troll comments can visually represent the negative judgments of their peers - for example by fading in colour or shrinking in size in proportion to the negativity of their score - but they should not disappear completely. (Note: it's reasonable to make an exception for obvious spam and/or libelous content and just delete it.)
Moderation ought to apply directly to comments, not to the users making them. People respond to incentives, and some trolls actually stop trolling and start participating responsibly when their trolls are voted down but their fair posts are voted up.
Similarly, blacklisting trollish users generally doesn't work. As long as it's easy to create sockpuppet accounts with throwaway email addresses, forums behave like badly applied wallpaper: when you push a bubble down, it just pops up somewhere else.

Censorship

Finally, a few words on censorship as an alternative method of blocking trolls:

Censorship is a slippery slope. Deciding which comments to delete is ultimately a judgment call, and people who are emotionally invested in the success of a forum are notoriously poor at remaining objective and applying standards consistently.

There's a risk that legitimate comments will get the axe. Even worse, there's a risk that an atmosphere of censorship will deter people from participating - the very problem that moderation was introduced to solve!

Finally, censorship gives the trolls strong rhetorical ammunition to accuse you of heavy-handedness and elicit sympathy from other participants. The perceived lack of transparency over which comments are acceptable puts everyone into an oppositional stance, and the trolls will be watching carefully for examples of hypocrisy.

Both philosophically and practically, censorship is at least as harmful as its targets. It depends naively on the consistent objectivity and benevolence of the moderators, introduces a chilling effect on participants who may hold legitimate beliefs that break with the majority, and arms trolls with a credible claim that their rights are being violated.

CouchDB Working Notes

2010-04-09T12:00:00Z

Note: this article is very much a work in progress. It functions mainly as a place where I can document what I learn about CouchDB as I play around with it and get a progressively better sense of what it does and how it works.

Summary

CouchDB is a non-relational database server built in Erlang that stores its data as JSON and communicates RESTfully over HTTP.

Install CouchDB

First, you need to install CouchDB. If you're using a civilized operating system with a package manager, this won't be difficult.

ryan@home ~$ sudo apt-get install CouchDB

The package manager will install CouchDB and all its dependencies. The installation script will finish with this hopeful line:

 * Starting database server CouchDB

That's it. Couch is up and running.

RESTful API

The first thing to know about CouchDB is that it that you connect to it over HTTP using the standard HTTP "verbs" like GET, PUT, POST and DELETE - the very same protocol that browsers use to connect to web servers.

This is important to understand: you don't need database drivers or clients or anything else to interact with CouchDB. You just need to be able to send an HTTP request to the CouchDB server and receive the response.

You can prove this by opening your browser and navigating to http://localhost:5984. It should return something like this:

{"CouchDB": "Welcome", "version": "0.10.0"}

So far, so good. But what are you looking at?

JSON

If you've built any Ajax web apps in the past few years, the output from the CouchDB web server should be familiar to you. It's JSON, or JavaScript Object Notation, the lightweight data format introduced by Douglas Crockford.

(BTW if you write JavaScript but haven't read Crockford's book JavaScript: The Good Parts, I highly recommend it. Chances are you're Doing It Wrong today.)

JSON is based on JavaScript object literal notation, a method of creating objects in JavaScript by literally describing their properties and methods:

myObject = {
    name: "Ryan McGreal",
    age: 36,
    emails: ["ryan@quandyfactory.com", "editor@raisethehammer.org"],
    doSomething: function() {
        alert("This is a useful method!");
    }
}

JSON supports arrays and dictionaries with nesting and various data types (number, string, boolean, array, object and null). It also supports schemas.

A major advantage over XML is that the syntax is much simpler and less verbose, which makes it lighter across networks as well as more human-readable.

While it's possible to evaluate JSON as JavaScript using eval(), it's not recommended (for what I hope are obvious reasons). It's much better to use a JSON parser.

The good news is that there are already JSON parsers for wide range of programming languages in addition to JavaScript. Python, for example, includes the json module as part of the standard library (as of version 2.6). There are also libraries for PHP, Ruby, Perl, Java, Scala, Erlang, Common Lisp, and Clojure, among others.

Create a CouchDB Database

Given that you connect to CouchDB over HTTP, it should not surprise you to learn that you execute actions against CouchDB using the HTTP request method 'verbs': GET, PUT, POST and DELETE.

As a result, any programming language that can communicate over HTTP can connect to a CouchDB server.

Let's create a new database. For simplicity's sake I'll connect to the CouchDB server on the command line using curl. The -X command line argument specifies which HTTP request method to use.

ryan@home ~$ curl -X PUT http://localhost:5984/mydb

CouchDB returns the following response (in JSON format, or course):

{"ok": true}

You now have a CouchDB database called mydb. Prove it by running a GET request against the URL for your database:

ryan@home ~$ curl -X GET http://localhost:5984/mydb

Check it out: your database has a URL!

CouchDB will return a summary of the database properties (I've added whitespace to the CouchDB JSON responses to aid readability):

{
    "db_name": "mydb",
    "doc_count": 0,
    "doc_del_count": 0,
    "update_seq": 0,
    "purge_seq": 0,
    "compact_running": false,
    "disk_size": 79,
    "instance_start_time": "1276950495049303",
    "dis_format_version": 4
}

Notice the doc_count property of 0. We should fix that by adding a document.

Add a Document

You add documents to a database using the HTTP POST method (the same method HTML forms use to send form data to the server).

ryan@home ~$ curl -X POST http://localhost:5984/mydb \
--header 'Content-Type: application/json' \
--data '{"title": "My First CouchDB Document", "author": "Ryan McGreal", \
"date_posted": "2010-04-09 11:38:26", "section": "blog", \
"content": "This is my first-ever CouchDB document. Niiice." }'

CouchDB returns a response like the following:

{
    "ok": true, 
    "id": "bf58234234aed2389dcb23423",
    "rev": "1-4ac239847fea987bd2340"
}

Let's take a look at what just happened.

We executed an HTTP POST request with a Content-Type header of "application/json" (since we're sending the data in JSON format) and a JSON object for the posted data. The JSON object includes the following keys and values:

title: My First CouchDB Document
author: Ryan McGreal
date_posted: 2010-04-09 11:38:26
section: blog
content: This is my first-ever CouchDB document. Niiice.

CouchDB accepted the JSON object, parsed it to ensure that it's valid JSON, and then created a document in the mydb database with the content you provided.

It also issued the document with an ID and a revision ID, which it included in the response so that you can access the document later.

The revision ID is additionally beneficial: you've got version control built into your database. CouchDB also serves revision numbers as HTTP ETags, so you can take advantage of caching.

View a Document

Let's take a look at the document we just created.

ryan@home ~$ curl -X GET http://localhost:5984/mydb/bf58234234aed2389dcb23423

Your document has a URL, too!

CouchDB returns the following response:

{
    "ok": true, 
    "id": "bf58234234aed2389dcb23423",
    "rev": "1-4ac239847fea987bd2340",
    "title": "My First CouchDB Document", 
    "author": "Ryan McGreal", 
    "date_posted": "2010-04-09 11:38:26", 
    "section": "blog", 
    "content": "This is my first-ever CouchDB document. Niiice."
}

There it is. You can access its properties and values just like you would a recordset.

The difference is that the CouchDB data is formatted in JSON and can be nested - i.e. an object's property could be an array or another object with its own keys and values - rather than forced into a flat table.

Add Another Document

Let's try adding a document with nested data:

ryan@home ~$ curl -X POST http://localhost:5984/mydb \
--header 'Content-Type: application/json' \
--data '{"title": "Some Handy Links", "author": "Ryan McGreal", \
"links": ["http://quandyfactory.com", "http://raisethehammer.org", \
"http://hamilton350.com"]}'

CouchDB responds:

{
    "ok": true, 
    "id": "e6ed2389dcb26100caeg3423",
    "rev": "1-7c21a0fd39fea987bd2340"
}

Now we have two documents in our database with different sets of fields - you're not forced to map your data into a standardized set of properties.

If we tried to do this in a traditional RDBMS, our documents table would have to have a links column with a Null value in the first document, as well as a content column with a Null value in the second document.

With CouchDB, you can give your documents arbitrary fields and values depending on what's applicable for each document.

Update a Document

[Coming Soon]

Data Views

[Coming Soon]

Misc.

Let's create another database.

ryan@home ~$ curl -X PUT http://localhost:5984/mydb

Wait a minute; didn't we just create a database called mydb?

CouchDB returns this response:

{
    "error": "file_exists", 
    "reason": "The database could not be created, the file already exists."
}

Whew, dodged a bullet there. CouchDB doesn't let you accidentally overwrite an existing database.

At any time you can get an array of all the databases on your server with the following GET request:

ryan@home ~$ curl -X GET http://localhost:5984/_all_dbs
["mydb"]

There's plenty more, but this should be enough to get you started thinking about how you might use CouchDB in projects whose data models don't necessarily lend themselves to strict relational structures.

Utils Web Interface

One last thing: CouchDB also provides a web interface with some handy utilities for navigating around the server and its databases.

Just load http://localhost:5984/_utils in your browser to see it.

MPAA and Piracy

2010-02-26T12:00:00Z

When I was a kid, I had a Fat Albert book in which one of the Cosby Kids (for details of who did what I'm going on memory, but I think it was Bucky) gets upset about something and runs away.

The other kids all search the projects for him, but evening is coming on and it's getting harder to see.

At one point, Weird Harold is walking from streetlamp to streetlamp, peering into the pool of light under each pool. Rudy walks up and asks Harold what he's doing. Harold says he's looking for Bucky.

But why are you looking under the streetlamps? asks Rudy.

Harold answers, That's the only place I can see.

Which brings us to:

(source)