TurnKey Linux Virtual Appliance Library

Why parallel programming is hard

Implementing Cloudtask took more time than I had planned due mainly to the challenges of parallel programming, which I hadn't done that much of before. Also, parallel programming really is inherently far more difficult than serial programming.

In my mind there are three major challenges:

Parallelize - a simple yet powerful high-level interface to multiprocessing

When I was developing Cloudtask, I discovered none of the interfaces in the Python multiprocessing module were powerful enough for my needs so I had to roll my own. The result is the generically useful multiprocessing_utils module in turnkey-pylib which from my totally subjective perspective provides a far superior interface to parallelization than the built-in multiprocessing interfaces.

TurnKey Hub: Not just adding random features

I started writing a review for cloudtask as a comment on the announcement, but decided it would be better to address a topic that was raised by Jeremiah when we launched the Hub API:

Introducing CloudTask - a cloud batch execution tool

The cloud. Isn't that just a new name everyone on the latest hype bandwagon is slapping on the same old stuff? Yes. Or rather, at least the way some clueless marketing types are using it that is. With so much smoke you'd forgive the cynics for thinking there's no fire. But... there are a few genuinely interesting things an IT guy can do today that just weren't practical a few years back.