TurnKey Linux Virtual Appliance Library

Introducing CloudTask - a cloud batch execution tool

The cloud. Isn't that just a new name everyone on the latest hype bandwagon is slapping on the same old stuff? Yes. Or rather, at least the way some clueless marketing types are using it that is. With so much smoke you'd forgive the cynics for thinking there's no fire. But... there are a few genuinely interesting things an IT guy can do today that just weren't practical a few years back.

Like renting an armada of servers you could never afford to buy to parallelize a computational task and get results in an hour instead of days, weeks or even months, for exactly the same cost. Now that's kind of exciting if you can pull it off.

We came across a modest version of this problem for various routine TurnKey related maintenance tasks such as rebuilding appliances. It sure was nice to be able to fire up a server on-demand, run a batch job and be able upload new images to sourceforge at 100MB/s. Usually we would leave the batch job running overnight and terminate the server the next day or so. That wasn't so bad for non-frequent tasks, but we realized we could do better. On Amazon EC2 running 10 servers for 1 hour costs the same as running 1 server for 10 hours. That's the theory anyhow. In practice launching and controlling many servers by hand can be painful.

Obviously, a bit of clever automation would be just the ticket. I hate reinventing the wheel so I first tried surveying existing solutions, but I couldn't find any that fit our needs.

So I rolled up my sleeves, and about a month or so later cloudtask was born. It's kind of neat. If you've ever had to do this sort of thing by hand or put together an ugly mess of scripts you'll probably find cloudtask an easier and more reliable primitive to build on.

Here's the documentation.

Here's a tacky video demo I whipped up (best viewed full screen):

You can get future posts delivered by email or good old-fashioned RSS.
TurnKey also has a presence on Google+, Twitter and Facebook.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)