TurnKey Linux Virtual Appliance Library

Amazon EC2: How to move data to instance storage

The problem: the 10GB root filesystem is running out of space

Here's an example to illlustrate the problem. Alon launches a Small type server via the TurnKey Hub. After a couple of months Alon notices that space on his root filesystem has almost run out.

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208   9082472    714448  92% /
none                    843604       124    843480   1% /dev
none                    880692         0    880692   0% /dev/shm
none                    880692        56    880636   1% /var/run
none                    880692         0    880692   0% /var/lock
none                    880692         0    880692   0% /lib/init/rw
/dev/sda2            153899044    192072 145889348   1% /mnt

The problem is that there is only 10GB in the server's root filesystem. The server has an additional 160GB of instance storage attached to /mnt.

So how does Alon move data to instance storage?

Limitations: instance storage is ephemeral, use EBS-backed volumes for persistent storage

Before you stop or destroy an instance, please make sure to backup any data you want to keep (e.g,. using TKLBAM).

Instance storage is guaranteed to be available only as long as an instance is running on the same physical host. That means you can rely on instance storage not to be deleted so long as you don't stop or destroy the server. You can still reboot because rebooting doesn't move an instance to a different physical server.

In other words:

For EBS-backed instances: instance storage survives reboots but doesn't survive if you stop an instance or destroy it. This is because stopping an EBS-backed instance detaches it from the physical host. When you restart your server (e.g., a year later) it will probably be resurrected on an entirely different physical host.

For S3-backed instances:  instance storage survives reboots but doesn't survive if you destroy the server.

Create EBS volumes: If you want to create storage volumes that survive a server being stopped or destroyed please create an appropriately sized EBS volume. Note that Amazon charges $0.10/month per GB for EBS volumes compared with instance storage which is free. Once you attach an EBS volume to a server you can move data from the root filesystem to the EBS volume as described below, except you'll be using a different path (e.g., /media/ebs/869c3f instead of /mnt).

Find out what is taking up space on your server

The ncdu program is a quick and easy way to find out what is eating up disk space on any Linux system. You can install and run it on TurnKey like this:

apt-get install ncdu
cd /
ncdu

As you can see in this example /home is our main disk space hog so that's the directory we will move to instance storage.

The solution: moving directories to instance storage

To move his home directory to instance storage Alon is going to copy it over to /mnt and then configure his server to bind /mnt/home to /home. Like this:

mv /home /mnt
mkdir -p /home
echo /mnt/home /home bind bind,nobootwait 0 0 >> /etc/fstab
mount /home

/etc/fstab is the file the server uses to keep tabs on the filesystem. We added a line to the fstab so that our filesystem reconfiguration survives reboot. We've added the nobootwait flag so that the machine boots even if due to some kind of error the system can't find /mnt/home. Otherwise the system could hang. As you can see above editing the file by hand is very simple but if you don't feel comfortable using the command line you can also configure filesystems using Webmin:

Go to: System -> Disk and Network Filesystems -> Add mount. Type: Loopback filesystem (bind)

Warning: Webmin doesn't move over your data when you reconfigure the filesystem. So if you already have data in /home you'll want to move it first:

mv /home /mnt
mkdir -p /home

Epilogue

After moving his /home directory to /mnt Alon once again has plenty of space on his root filesystem:

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208    714448   9082472   8% /
none                    843604       124    843480   1% /dev
none                    880692         0    880692   0% /dev/shm
none                    880692        56    880636   1% /var/run
none                    880692         0    880692   0% /var/lock
none                    880692         0    880692   0% /lib/init/rw
/dev/sda2             153899044  8560097 137521323   5% /mnt

Comments

Thanks

Thanks, this was exactly what I needed to know. I think you should point out that you're running as root, especially since funny things can happen if you're still a regular user when your /home gets moved away (which doesn't happen to root, because its home is /root). I didn't have any serious problems though.