TurnKey Linux Virtual Appliance Library

Launching TurnKey Hub into private beta: cloud deployment simplified

Towards the end of last year we decided it was time to start working on an idea we've been toying with for a while. Mapping out the feature set was fun, and a lot of the current and future features are based on feedback we received from you guys and gals, as well as many related questions and comments from around the net.

Today after a last round of internal testing we are pleased to announce that we've launched TurnKey Hub into private beta:

https://hub.turnkeylinux.org

We've just sent out the first batch of invites. If you've already requested an invite, you should receive one as we roll them out. Please be patient. We initially have limited capacity so it's first come, first served.

What is this TurnKey Hub?

  • The short version: TurnKey cloud deployment: simplified.
  • The slightly longer version: An easy way to launch and manage TurnKey Linux appliances in the Amazon EC2 cloud (more clouds and VPS providers on the way).

A simple Amazon EC2 console optimized for TurnKey

Hub Servers

  • Launch TurnKey appliances in the click of a button.
  • TurnKey optimized firewall templates.
  • Configure custom passwords on launch.
  • Authenticate with personal SSH key in addition to EC2 keypairs.
  • Automatically setup EBS devices and Elastic IP's on launch.
  • Easier management with descriptive labels for all assets.
  • Unified interface for all regions and all your EC2 accounts.

And that's just the tip of the iceberg. There's much more in development...

Upcoming features

  • More clouds: Support all clouds and VPS providers.
  • Backup: Automatic encrypted appliance backups.
  • Migration: Automatically restore backups anywhere.
  • You decide: Suggest features and help us prioritize.

In other words this is just the first modest step in a much more ambitious plan to continue making TurnKey easier to use, as Liraz recently explained:

"Imagine being able to develop your site on a locally running appliance (e.g., running in VirtualBox or VMWare). Then, when you're ready you can automatically migrate your appliance, with all your customizations to a cloud hosting provider of your choice."

So once you receive your invite, take the Hub for a spin, let us know what you think. We'd like to know how to make this better. What new features you'd like to see implemented. That sort of thing.

Remember, you can request an invite here.

You can get future posts delivered by email or good old-fashioned RSS.
TurnKey also has a presence on Google+, Twitter and Facebook.

Comments

David Hall's picture

The Hub is cool!

Thanks Turnkey Gang for a very very easy to use Virtual Appliance deployer!

In less than 30 minutes, I created my Amazon EC2 account, setup a Media Wiki and was off and running.

I'm sure there is more to work out - ie. something tells me that when I terminate the host, the data goes with it.  Or is the state preserved / snap shotted.

When does one need to use S3? 

All that can be worked out later.  I'm impressed!  I must say it makes me nervous to put my Amazon keys into your hub as that gives one access to run up a fierce bill..  What makes Hub Secure enough to do this?

David Hall @ The Agile Architect dot com

Liraz Siri's picture

Data persistence and Security of the Hub

Currently, if you want your data to survive instance termination you should store it in an EBS volume. But it's not automatic.. yet. The next major feature we'll add into the Hub is automatic backup and migration which will take care of that usage scenario and many others.

Regarding security, that's a good question. I think it would make a good idea to dedicate a blog post to this but I'll cover the main points right now:

  • Security background: We, the developers, come from a computer security background. You can Google us to find the full details. For example, Alon, which is the lead developer for the Hub founded a computer security training company. Before that he worked as a penetration tester, and before that he was responsible for network security in the Israeli air force. My background is similar. Security is a big priority for us and we are not naive regarding the risks.
  • High-level language and webapp framework: The Hub is implemented in a high-level language (Python) using the Django framework.  which contains a range of security mechanisms you can use to protect from all sorts of common web application attacks (e.g,. SQL injection, XSS, XSRF, etc.). Using a high-level language wipes out a large class of security vulnerabilities (e.g., issues with out-of-bound pointers, etc.)
  • Encryption: All of your network traffic to and from the Hub is encrypted.
  • Isolation: We've setup the Hub to run in a separate server from the rest of our network infrastructure. It's isolated.  If the CMS that runs this web site is compromised, or our mail server, etc, that doesn't effect the Hub.
  • Security audits: We've run extensive security audits on the Hub prior to launch including manual code inspection AND a gauntlet of automatic tools.
  • Reputation: We have a reputation to maintain and understand the consequences of failure. If the Hub's security fails that will be a huge setback for the viability of the service. This is a risk we take very seriously.

The precautions we're taking, coupled with our background mean that it's far more likely in my opinion that your access keys / credit card / bank account credentials will be stolen via  malware running on your PC (for example). Nothing is perfect but if you already accept the risk of doing any sort of commerce on the Internet you shouldn't loose any sleep over the Hub storing your AWS keys.

Note that Amazon allows you to create a separate set of Amazon keys you can use just for the Hub if you believe this reduces your risk. You can then revoke the keys at any time.

Will it be opensourced?

Will it be opensourced?

Liraz Siri's picture

Needs more thought

There are pros and cons either way. The Hub isn't a generic web application that it would make sense to create multiple instances of. It's a complex piece of infrastructure that is tightly integrated with the TurnKey Amazon Machine Images. If I've learned anything doing TurnKey is that open source is harder than it looks. It's not magic and it seems to work best when you have a repeatable, generic need and it's both easy and desirable for outsiders on the same side of a problem to jump in and collaborate.

I'm not yet convinced that's the case with the Hub. As we add more features it's going to grow more complex and become more tightly integrated with the TurnKey images.

But eventually once things have settled down maybe we can extract some generically useful pieces out of the Hub and open source that. Needs more thought...

Running on private clouds

I think there may be a repeatable, generic need for cloud owners.

Open source cloud systems like Eucalyptus and OpenNebula are compatible with the Amazon API so using the TurnKey Hub on these systems should not be very hard. And having ready to run appliances in your own cloud is a real benefit.

I'd probably use it and contribute if it was open source.

opennebula

I think opennebula's AWS compatibility is more limited than Eucalyptus (or Canonical's UEC).

From what I know OpenNebula is like OpenQRM in that there is a plugin type architecture that allows a certain subset of Virtualization support ... create vm, delete vm, attach/detach storage etc.

UEC and AWS are probably the closest in terms of API

Migration from Private cloud to Amazon

Hi Guys,

I've been using your Drupal app for my blog for about a year now and I love it. I've upgraded and customized it as I've gone along, and it kinda helped me ease into the Linux world (I'm a reform-ing MS server guy).

I currently run the Drupal appliance on my own ESX 4 box using a DSL connection, but now would like to migrate it to EC2.

Do you have anything in place for this yet or do you know of a 3rd party way of doing it effectively?

Regards,

Mike

Liraz Siri's picture

Backup and migration functionality in development

We're developing a secure backup and migration tool that will allow you to seamlessly do exactly what you want. The back-end is working "in the lab" but we haven't yet finished developing the front-end for it. This is the next major feature we will be integrating into the Hub though I can't give you a hard schedule for when it will be available so if you need this done right away you may want to look into migrating your Drupal instance manually. That can be a pain but that's how everybody does it right now. Basically you'll need to transfer your theme, contrib modules and database. The "Backup and migrate" Drupal module can help you with the database. You might want to experiment doing this between two VMs in your ESX setup first...

Migration from Private cloud to Amazon

Now THATs a tool I look forward to.

Okay, looking at the Backup and Migrate plugin now.

Is it advisable to just transfer entire directories (/etc/drupal/**   and /Var/***) where basically anything Drupal goes, or just really focus on the Theme, and Modules directory? And the DB of course.

Liraz Siri's picture

I'd focus on the theme and modules

Bulk copying directories can have unexpected results. That might work too mind you, but I'd test it carefully first. When you copy your theme and modules you know exactly what you're in for. What I've done with this website (which is also in Drupal) is sync the VM devel box's themes and modules to the production box. Initially you also sync the database but after your site is up and running you'll usually want to sync the database from the production site to the development VM and not vice versa.

Also, you may want to read this blog post.

Migration from Private cloud to Amazon

Got it. Read the post and I have a much better idea of what I'm doing now!

Thanks for the quick help and detailed replies.

Liraz Siri's picture

In case anyone missed it TKLBAM has been out for a couple weeks

Just remembered this thread. I'm sure (cough) that you are all subscribed to the TurnKey Blog, but in case you're not the announcement is here.

ESX to AWS EC2

There are several paths you can follow to convert one of your ESX VMs to run on AWS EC2.

If your drupal system is less than 10G in size you can use the AWS EC2 AMI tools create a copy of your whole machine and move it to AWS (S3) the run an instance of it on EC2

To translate virtual-virtual you first need to get your VMware vm into a flat file format using either VMWare converter or QEMU.

Once you have the flat file you can use the AWS AMI tools to:

bundle the image (ec2-bundle-vol command)

then the bundled image and upload to AWS S3 storage (one command: ec2-upload-bundle)

then register your new AMI (1 command) ec2-register in order to be able to launch your instance

This way you have a complete snapshot of your current system

But if your data makes the created image too big (10G) you could just bundle/upload/register your base machine then later FTP (or use the firefox add-on called S3Fox)  S3Fox is a companion to the Firefox AWS mgmt tool called ElasticFox... both of which you should learn more about as they are great  tools for working with AWS's cloud.

Using S3Fox uploading your created image to AWS S3 is a simple click of the mouse (after you've cofigured S3Fox with your AWS Credential and specified a S3 storage "bucket" name to store your image into.

Now the above will give you what is called an S3 backed image... where your root file system is limited to 10G ... but you can mount other storage.

However, today most people are creating and using what are called EBS back AMIs.    Elastic Block Storage (EBS) AMI's boot faster and can be started/stopped (ie paused) without terminating the instance.  With EBS backed AMI you aren't limited by root file system and your data is persistent on the EBS volume which by the way you can make snapshots of whenever you make significant changes.

I know that's alot to through out there but hope it helps a bit.

 

 

 

how about an Open Source Alternative to Canonical's Landscape

Growth in private clouds based on canonical/ubuntu's UEC/eucalyptus based cloud architecture is on fire.  

I've signed up and used Canonical's Landscape application and it works well and it does let you manage both your private UEC cloud as well as AWS VMs.... but I think their pricing of it makes it prohibitive (at least to me) as there is a subscription cost for every machine/vm managed.    Which can climb in $ rapidly.

I'd like to see an alternative Open Source application that provides an intelligently integrated combination of something like the core capabilities of:

  • ebox - for network, security, ddns, printer, AD, etc. mgmt/reporting
    • ebox already integrates a great deal of this from various GPL projects
  • integrated with -- chef/puppet/cfegine - some provisioning mgmt tool with a Gui front-end?

I think in a cloud environment, people would be willing to pay a decent subscription for this type  of cloud manager application.  

I think Canonical's UEC marketing plan approach requirement to pay for cloud controller, cluster controller, storage controller, node controller ... and ... vms is too much.

Second..  But don't you just

Second..  But don't you just hate it when you throw an idea out there and you have no clue if it was dropped, or if someone is making a business plan this very moment to make millions.. er billions off your excellent idea..

Amazon over capacity

Hello! I have been trying to launch a Drupal 6 appliance to Amazon EC2 via the Hub and I keep getting the message that Amazon is over capacity. Got it last night and just now too. Anyone know what's going on?

Different Region?

Did you try a different region?

Alon Swartz's picture

Amazon regions are split into zones

Amazon regions are split into zones. When you launch an instance in the AWS console and you don't specify a specific zone (which is the default), Amazon will launch your instance in the "best" zone available.

But, that comes with a cost as some assets, such as EBS volumes, are zone specific. So if you have an EBS volume in us-east-1a but your instance is launched in us-east-1b, then you can't attach the volume to your instance. For this reason, you can force a specific zone.

When launching instances via the Hub, it will force the first zone (e.g., us-east-1a), and when you create EBS volumes they are created in the first zone (e.g., us-east-1a) as well. This was a design decision to keep the user interface an usage as simple as possible. There is a downside though - at certain peak times Amazon is over capacity and you won't be able to launch new instances until they have provisioned more servers.

Bottom line, you can either wait a couple of minutes or specify a different region.

Up until now there hasn't been too many capacity issues, but if the persists (yes, I know, its really annoying) then we'll have to see what we can do about it.

Still over capacity?!

Keep getting this message when trying to launch an instance from turnkeyhub to us-east-1 (N. Va.)

"Amazon EC2 is temporarily out of capacity.Try launching in a different region or wait a few minutes."

It's 8am EST on a Monday.  Doesn't make sense.  Sure a work-around is to try the us-west region but AWS pricing is more expensive for that region. 

Why the constraint for the east via turnkeyhub?


Alon Swartz's picture

AWS insufficient capacity issues should be solved

I just deployed an update to the Hub which should fix the insufficient capacity issues.

As I explained above, prior to this update the Hub would default to and force the first zone (e.g. us-east-1a) in each region to simplify the user experience with use of EBS volumes. This update changes that. There is no global, default, hardcoded zone anymore.

The Hub will now use a default_zone for each region according to your prior use of the region. For example, if you have an EBS volume in us-west-1b, when you launch an instance in us-west-1, us-west-1b will be forced so you can attach the volume.

Another example, if you already have an instance registered in the Hub in eu-west-1a, when you create an EBS volume in eu-west-1, it will automatically be created in the correct zone.

And finally, if you don't have any instances or volumes in a region, the Hub will request that Amazon choose the best zone when launching an instance in that region. The zone Amazon chooses will become the default zone for that region, until such time that the instance and (if applicable) volumes are deleted from the Hub.

Closing note, if you don't know what zones are, ignore this post - the Hub will do what is best, you don't need to worry about it :)

Amazon over capacity... I just checked... don't think so

I don't think its amazon as I've been bringing up/down instances for the past 2 days.

Have you tried to launch any AMI's from the AWS Console instead just to see if the Instance starts?


 

tried a new zone

I tried US-west-1 instead of us-east-1, and it worked. Simple solution, thanks for the help.

adding an existing Turnkey AMI

I have an existing Turnkey Instance running, can I add this to the Cloud app so I can view/manage it?

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)