Introducing CloudTask - a cloud batch execution tool

The cloud. Isn't that just a new name everyone on the latest hype bandwagon is slapping on the same old stuff? Yes. Or rather, at least the way some clueless marketing types are using it that is. With so much smoke you'd forgive the cynics for thinking there's no fire. But... there are a few genuinely interesting things an IT guy can do today that just weren't practical a few years back.

Like renting an armada of servers you could never afford to buy to parallelize a computational task and get results in an hour instead of days, weeks or even months, for exactly the same cost. Now that's kind of exciting if you can pull it off.

We came across a modest version of this problem for various routine TurnKey related maintenance tasks such as rebuilding appliances. It sure was nice to be able to fire up a server on-demand, run a batch job and be able upload new images to sourceforge at 100MB/s. Usually we would leave the batch job running overnight and terminate the server the next day or so. That wasn't so bad for non-frequent tasks, but we realized we could do better. On Amazon EC2 running 10 servers for 1 hour costs the same as running 1 server for 10 hours. That's the theory anyhow. In practice launching and controlling many servers by hand can be painful.

Obviously, a bit of clever automation would be just the ticket. I hate reinventing the wheel so I first tried surveying existing solutions, but I couldn't find any that fit our needs.

So I rolled up my sleeves, and about a month or so later cloudtask was born. It's kind of neat. If you've ever had to do this sort of thing by hand or put together an ugly mess of scripts you'll probably find cloudtask an easier and more reliable primitive to build on.

Here's the documentation.

Here's a tacky video demo I whipped up (best viewed full screen):


Jeremy Davis's picture

We had another vid that was removed by them too, but luckily it was also on YouTube. I just had a quick google and couldn't see this one there though... :(

I have just emailed Alon and Liraz and hopefully one of them still has the original vid and can re upload it to YouTube.


Add new comment