For the project we are working on for Teachers Without Borders we need a reliable asynchronous way to process tasks. There are a myriad of ways to do this and if you aren't careful you'll spend weeks second guessing yourself while investigate every possible option. Not that I would know that from experience.

We chose Starling and Workling as our solution. Starling is a message queue that uses memcache. It was built by the guys at Twitter and released as open source. I bet it scales.

Workling is a great plugin built by a couple of guys working on boomloop.com It makes using Starling a bit easier especially when it comes to consuming messages from the queue.

There are a few blog posts around that describe how to get stuff up and running. Other than that the documentation is a bit sparse. I have to believe that will change over time. I am throwing my own links and experience into the mix so I don't forget how to do this.

First get the gems you need to run starling:
sudo gem install starling
sudo gem install memcache-client

Get the working plugin
git clone http://github.com/purzelrakete/workling/tree/master/Changes

Get starling and working going with:

Fire up starling - this isn't in daemon mode. Add a -d for that.

starling -P tmp/pids/starling.pid -q tmp/starling

Next fire up the server

script/server

Get workling going

script/workling_starling_client start

A few more resources

http://www.williambharding.com/blog/rails/starling-update-from-the-horses-mouth/

http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/