TurnKey Linux Virtual Appliance Library

New Hub feature: Server snapshots

I usually get excited when adding new features to the TurnKey Hub. Recent excitement included server monitoring, reserved instances, domain management, and the Hub API.

I'm very excited about todays annoucement, not only is it awesomely useful, it's also technically cool!

So what are snapshots?

I'm sure you can guess, but let me explain anyway.

Snapshots can be used with EBS-backed instances to create point-in-time snapshots of the root filesystem, which are persisted to Amazon S3 for storage durability. Snapshots are incremental, meaning that only changes since the last snapshot are saved, taking up less storage, time, and reducing costs (see below for technical details).

Snapshots ask Amazon's fiber-optic storage backplane to save your server's disk state while it's running without impacting performance.

Ok, but what can I do with them?

Server clones

Snapshots can be used as the basis for a new server, essentially creating a clone (the cloud server equivalent of a time machine crossed with a portal to a less obnoxious alternative dimension), for example:

  • You can clone a production server to create a staging enviroment for testing new features, hacking away, whatever, without the worry of hosing your production server (guess how I tested this new feature).
  • You can essentially upgrade your servers hardware if you need the extra horse power, memory or even disk space. Say you were testing an idea with a micro instance, and now its taking off. Firstly congrats, secondly just clone the micro's latest snapshot to a larger instance size and update the DNS record / re-associate the elastic IP.
  • Let you're imagination run wild!

EBS Volumes

Snapshots can be used as a starting point for a new EBS volume, for example:

  • You mistakenly deleted a file, hosed your database, or whatever bad thing that can happen. You create a volume from the snapshot of your choice, attach it to your instance (which is auto-mounted via ebsmount) and access the data you need.
  • Again, let you're imagination run wild!

Can I schedule automatic snapshots?

You sure can! You can schedule automatic zero-load server snapshots for hourly, daily, weekly and monthly frequency, or manually create one at anytime.

There is however a snapshot limit per Amazon account, per region, so when configuring automatic scheduled snapshots, snapshot retention is also configurable to prune old snapshots, keeping you within the limit and saving you money.

Sounds cool, what does it look like?

We've added 2 new fields to the server record:

Snapshots - Server Record

And this is the snapshot dashboard:

Snapshot - Dashboard

Are there any limitations?

Snapshots only support EBS-backed instances, and not S3-backed instances. This is a technical limitation as snapshots are performed on the EBS-backed root volume, which S3-backed instance do not have.

Snapshots are saved to S3 storage, but they will not appear in your S3 buckets, nor can you access them using the standard S3 API. To access snapshot data you need to create an EBS volume or a server clone.

As mentioned above, there is a limit of the amount of snapshots each Amazon account can have, but you can request to increase your limit (specify you want the snapshots limit increased in the comments.)

Data consistency: Do not solely rely on snapshots for backups, as they may become inconsistent due to disk-buffering and locking. We use TKLBAM for our backups, and suggest you do the same.

Technical details - snapshots explained

I mentioned that snapshots are technically cool, and that they are incremental - let me try and explain what that means at how it works behind the scenes.

A snapshot of an EBS volume can be taken at anytime, which asks Amazon's fiber-optic storage backplane to save the data stored on the volume, at the block level, at that exact point-in-time, to S3 storage.

To improve performance and reduce storage space, Amazon will only copy the blocks of the volume that have changed since your last snapshot - hence incremental.

Now for the extra cool part, unlike regular incremental backup chains, you can delete any previous snapshot. Huh? What? Yep, snapshots are not chained, but are rather conceptually like a table-of-contents of pointers to saved data blocks.

When you delete a snapshot, only the data blocks that are solely used by that specific snapshot are deleted. Data blocks that are used by subsequent snapshots are not. In the below illustration, if SNAP-B is deleted, only SNAP-B:block-2 will be deleted from Amazon S3 as a newer version (SNAP-C:block-2) has already been saved.

Snapshots - Blocks


Bottom line, take snapshots for a spin and let us know what you think.

You can get future posts delivered by email or good old-fashioned RSS.
TurnKey also has a presence on Google+, Twitter and Facebook.

Comments

Jeremy's picture

Nice one Alon!

Looks good mate. Another nice complement to the TKL/Hub lineup!

L. Arnold's picture

Very Interesting.. I use these in ESXI and will be great here!

I need to spend some time testing this...  I ran a snapshot yesterday then tried to build a new Cloud but I think I missed some elements...  Once I get the method down though I expect it will be extremely helpful..  I already noticed how fast this will build a new server.

As always, learning the protocol is where to start. 

Thanks for the hard work!

L. Arnold's picture

Slightly different process but able to build new server frm Snap

In VMware you "revert" to a Snapshot.  In TKL Hub, you build a new server from snapshot.

ONe command then needs to be issued.

IP Address generally needs to be reassigned.

it works,,, or did so today anyway quite nicely.

Chris Musty's picture

Benefits over TKLBAM

I have just read through very quickly and either I have missed soimething or I am thick - aside from ease of use what is the benefit over TKLBAM?

Also is Postgres supported from snapshots?

Chris Musty

Director

Specialised Technologies

Jeremy's picture

From my understanding

This new 'snapshot' feature is fundamentally different to TKLBAM in that it is a whole filesystem image (think 'snapshot' in VBox or VMware) and thus PostgreSQL (and any end user modifications/inclusions/exclusions) are supported without any technical knowledge or tweaking required.

So whilst it's not a replacement for TKLBAM it could be an easy way to 'clone' existing instances and/or revert to a previous state. I guess one downside would be that the size would be significantly larger (compared to TKLBAM). One obvious upside (as mentioned above) is that server customisations are included without need to tweak anything.

Snapshot Vs backup

I have some questions that may be not related to the topic discusses here, but as per my understanding anyone who follow this blog, would be interested to know the answers of these questions. So firstly I would ask sorry to take the conversation in different direction.

 

As I understand, Snapshots are  point-in-time copy of data that you have on disk, and backups are exact copy of data on disk (that must be point-in-time for some point of time). Snapshot storage take very less space in backup media and backup disk take very large space( About equal amount of space of data) in backup media. So Why don't we use snapshot as a replace of backup? Using this we can save a lot of space in backup media as well the time taken in backing up and restoring data from backup will be saved.

 

I am new to backup and storage concepts, so if I did missed any concept regarding backup and snapshot then I pardon sorry.

 

Thanks,

Abhinav Chittora

Jeremy's picture

You've got it around the wrong way...

Snapshots are whole of disk. TKLBAM backups are req'd data only (and thus much smaller). Both snapshots and TKLBAM backups can be incremental (ie just save the changes between the original backup and the current machine state).

That is why snapshots aren't a replacement for backups! Although snapshots can be very handy, they are not a replacement IMO.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)