I'm impressed with the way all the pieces of TurnKey just fit together like puzzle pieces. Even as new functionality is added it never seems like something gets patched onto the existing framework. Instead, it just integrates smoothly with what's already there.
While the new Hub features are coming out incrementally, they really are part of a bigger unifying vision. I don't like to hype up vaporware, but we have a lot of great ideas we're working on and we like to think we're just getting started. There's a big gap between what the Hub does right now and what we envision it doing in the future.
Also note that we're using the Hub internally to help develop and test TurnKey, so we're probably using it more than the typical user and run into its limits sooner. We're scratching our own itch, and the community benefits from that.
To recap, some of the features Liraz was referring to in his comment, and some that came after his comment are:
- Hub 1.0 follow-up: improved server status
- New Hub feature: Auto-Restore TKLBAM backup to a new cloud server
- Announcing public API for TurnKey Hub
- TurnKey Domain management & Dynamic DNS
- TurnKey 11.2, free micro instances, EBS backed cloud servers
- Introducing CloudTask - a cloud batch execution tool
More features are coming, but this a great milestone to stop, reflect, and talk a little about a pain point we've had, and how the features we've released to this point are helping us solve it.
Building and maintaining exponentially growing library
TurnKey currently has 45 appliances, available in ISO, VMDK, OVF, EC2 S3 and EBS backed, Xen and Eucalyptus.
To sum up, thats 585 images we need to build and maintain.
The (long overdue) TKL 11 part 2 will double the appliance library. Additionally we are working on supporting more build targets, as well as the new Tokyo EC2 region. Throw 64bit support into the equasion and the number of images to build and maintain will grow to about 3,300!
Automation and a glimpse into a pain point
We try to automate everything we do, but even executing and managing the stuff thats already automated needs automation when you reach scale.
To give you a little glimps into what I mean. When building the Amazon EC2 S3 backed images (in the past), I couldn't do it locally as my upstream kinda sucks are would take me about 2 weeks to build and upload.
So do it in the cloud, right? Correct. But building and uploading from an EC2 instance still takes a couple of days, mainly due to subsequent builds and uploading images built for different regions is quiet slow.
OK, so lets just spawn more servers in the cloud, in the different regions. Yep, this allows us to perform the relative region builds so upload is much faster, and they run in parallel.
But manually launching each instance from the Hub's web UI (pre- hub-api), updating my /etc/hosts file to keep track of the different instances (pre- hub-domains), logging into each instance and transfering the build infrastructure (pre- restore-on-launch), and finally starting the automated build process while making sure nothing breaks is painful to say the least.
It would also be much faster (and cost the same) if we could split the appliance builds amoungst more instances in each region, but try managing that manually and keep your hair - argh!
To throw some salt on an already bloody wound, remember we also have to build the other target formats (EBS, VMDK, OVF, etc...). Try do that without going bald.
Parallel batch execution with auto-launched cloud servers
This is where integrating the features mentioned above, and cloudtask comes into play.
Instead of pulling my hair out and waiting days, yesterday I threw a couple of tasks at cloudtask, which launched about 50 instances spanning the globe, our build infrastructure was automatically setup on each instance, and cloudtask started dishing out jobs to each of them.
Within about 1 hour, I received reports via email that all the builds completed successfully and were published. Pain point gone.
Now thats automation on steriods! And best of all, I get to keep my hair!
As mentioned above, we are just getting started. Lots more to come - it's getting exciting...