TurnKey Linux Virtual Appliance Library

Amazon EC2: How to move data to instance storage

The problem: the 10GB root filesystem is running out of space

Here's an example to illlustrate the problem. Alon launches a Small type server via the TurnKey Hub. After a couple of months Alon notices that space on his root filesystem has almost run out.

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208   9082472    714448  92% /
none                    843604       124    843480   1% /dev
none                    880692         0    880692   0% /dev/shm
none                    880692        56    880636   1% /var/run
none                    880692         0    880692   0% /var/lock
none                    880692         0    880692   0% /lib/init/rw
/dev/sda2            153899044    192072 145889348   1% /mnt

The problem is that there is only 10GB in the server's root filesystem. The server has an additional 160GB of instance storage attached to /mnt.

So how does Alon move data to instance storage?

Limitations: instance storage is ephemeral, use EBS-backed volumes for persistent storage

Before you stop or destroy an instance, please make sure to backup any data you want to keep (e.g,. using TKLBAM).

Instance storage is guaranteed to be available only as long as an instance is running on the same physical host. That means you can rely on instance storage not to be deleted so long as you don't stop or destroy the server. You can still reboot because rebooting doesn't move an instance to a different physical server.

In other words:

For EBS-backed instances: instance storage survives reboots but doesn't survive if you stop an instance or destroy it. This is because stopping an EBS-backed instance detaches it from the physical host. When you restart your server (e.g., a year later) it will probably be resurrected on an entirely different physical host.

For S3-backed instances:  instance storage survives reboots but doesn't survive if you destroy the server.

Create EBS volumes: If you want to create storage volumes that survive a server being stopped or destroyed please create an appropriately sized EBS volume. Note that Amazon charges $0.10/month per GB for EBS volumes compared with instance storage which is free. Once you attach an EBS volume to a server you can move data from the root filesystem to the EBS volume as described below, except you'll be using a different path (e.g., /media/ebs/869c3f instead of /mnt).

Find out what is taking up space on your server

The ncdu program is a quick and easy way to find out what is eating up disk space on any Linux system. You can install and run it on TurnKey like this:

apt-get install ncdu
cd /

As you can see in this example /home is our main disk space hog so that's the directory we will move to instance storage.

The solution: moving directories to instance storage

To move his home directory to instance storage Alon is going to copy it over to /mnt and then configure his server to bind /mnt/home to /home. Like this:

mv /home /mnt
mkdir -p /home
echo /mnt/home /home bind bind,nobootwait 0 0 >> /etc/fstab
mount /home

/etc/fstab is the file the server uses to keep tabs on the filesystem. We added a line to the fstab so that our filesystem reconfiguration survives reboot. We've added the nobootwait flag so that the machine boots even if due to some kind of error the system can't find /mnt/home. Otherwise the system could hang. As you can see above editing the file by hand is very simple but if you don't feel comfortable using the command line you can also configure filesystems using Webmin:

Go to: System -> Disk and Network Filesystems -> Add mount. Type: Loopback filesystem (bind)

Warning: Webmin doesn't move over your data when you reconfigure the filesystem. So if you already have data in /home you'll want to move it first:

mv /home /mnt
mkdir -p /home


After moving his /home directory to /mnt Alon once again has plenty of space on his root filesystem:

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208    714448   9082472   8% /
none                    843604       124    843480   1% /dev
none                    880692         0    880692   0% /dev/shm
none                    880692        56    880636   1% /var/run
none                    880692         0    880692   0% /var/lock
none                    880692         0    880692   0% /lib/init/rw
/dev/sda2             153899044  8560097 137521323   5% /mnt


Jeremiah's picture

Clearing up Ephemeral and EBS storage confusion


Instance storage (also called ephemeral storage) exists for every single EC2 instance even if it's not used.  It comes free of charge.  It can survive a reboot but will NOT survive a stop (shutdown) or destruction of the instance.

Instance store backed instances just mean that the root partition is stored on this ephemeral storage.  The vulnerability of this data being lost upon an instance shutdown or destruction means you should either take precautions to backup the data regularly or just decide that you don't care about it.


EBS volumes will always persist (survive) the reboot, stop (shutdown), or destruction of an instance.  

EBS backed instances use an EBS volume to store the root partition.  That means you can be confident that your root partition data will not only survive a reboot of the instance but that it will also survive a stop (shutdown) or destruction of the instance.

The same can be said of any regular EBS volumes you may add to an instance after startup.

EBS backed instances also have instance storage available to them but normally it is not exposed (mounted) to the operating system by default.  Apparently TurnKey Linux AMI's automatically mount this instance storage to /mnt making it readily available for use.

I think this particular document really needs to make a warning more clear that while you are able to move data to the instance storage you are making that data vulnerable to complete loss if you ever stop (shutdown) or destroy the instance.  So being able to move your /home directory may be the right solution for you either temporarily or permanently but just be aware that you will lose all that /home directory data if you ever stop (shutdown) or destroy your instance.

Most of this information can also be found in Amazon's Summary of Differences documentation.

Guest's picture


Thanks, this was exactly what I needed to know. I think you should point out that you're running as root, especially since funny things can happen if you're still a regular user when your /home gets moved away (which doesn't happen to root, because its home is /root). I didn't have any serious problems though.