Extending an LVM volume: Physical volumes (partitions) -> Volume groups -> Logical volume -> Filesystem
By Liraz Siri - 33 comments | Latest by spaceyjaseLogical Volume Management (AKA LVM) is a powerful, robust mechanism for managing storage space.
In TurnKey 11, instead of installing the root filesystem directly to a fixed size partition, we setup LVM by default, and install the root filesystem to a Logical Volume, which may later be expanded, even across multiple physical devices.
Unfortunately, as with anything powerful, to get the most out of LVM you first have to negotiate a learning curve. From the feedback we've been getting it seems that confusion regarding LVM is common with new users, so here's a quick "crash course"...
How LVM works
In LVM, there are several layers, each builds on top of the other:
PV[s] (Physical Volumes) -> VG[s] (Volume Groups) -> LV[s] (Logical Volumes) -> Filesystems.
Logical Volumes are allocated/extended within the boundaries of their underlying storage pool which is called a Volume Group in LVM terminology.
For example, in TurnKey the filesystem is installed by default to the /dev/turnkey/root Logical Volume, which is allocated within the turnkey Volume Group:
--- Logical volume --- LV Name /dev/turnkey/root VG Name turnkey LV UUID OzX3fe-aRQa-81XM-0vCV-8aJo-eUL4-6J90XJ LV Write Access read/write LV Status available # open 1 LV Size 17.0 GiB Current LE 4502 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 251:0
Out of the box the turnkey Volume Group doesn't have too much free space:
# vgdisplay --- Volume group --- VG Name turnkey System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 1 Act PV 1 VG Size 18.14 GiB PE Size 4.00 MiB Total PE 4645 Alloc PE / Size 4480 / 17.50 GiB Free PE / Size 165 / 660.00 MiB VG UUID IwaFL0-QCi8-iIUE-TWjQ-R906-PYpH-gMPaH9
We can only extend a Logical Volume within the free space of the underlying Volume Group. How much free space we currently have within the Volume Group can be seen in this part of the output:
Free PE / Size 165 / 660.00 MiB
In the above example we only have 660 MB to allocate to LVMs within the turnkey Volume Group. So if we want to extend the root LV we'll have to first extend the VG backs it up.
Volume Groups group together Physical Volumes. That's why they're called Volume Groups. This command will show us which Physical Volumes have been registered into LVM, and to which volume groups they have been assigned:
# pvdisplay --- Physical volume --- PV Name /dev/sda2 VG Name turnkey PV Size 18.15 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 4645 Free PE 165 Allocated PE 4480 PV UUID H1Prpv-0VXR-7moE-zsbt-eyVt-m0fQ-GkAT6w
In this example we only have one Physical Volume (the /dev/sda2 partition) in the turnkey Volume Group.
Extending a Logical Volume
Bottom line: if the underlying Volume Group doesn't have enough free space, to extend the Logical Volume you'll first have to extend the underlying Volume Group by adding another Physical Volume to it.
In VMWare you could either create a new virtual hard disk device to add to the volume group, or extend an existing virtual hard disk device, create a new partition with cfdisk, and add the new partition to the Volume Group:
# example #1: you've added to VMWare a new virtual hard disk called /dev/sdb pvcreate /dev/sdb vgextend turnkey /dev/sdb # example #2: you've expanded the existing sda hard disk cfdisk /dev/sda # creating /dev/sda3 (you may need to reboot before you can see this) pvcreate /dev/sda3 vgextend turnkey /dev/sda3
After you've extended the Volume Group, you are free to extend the underlying Logical Volume:
# lvextend -L+10G /dev/turnkey/root Extending logical volume root to 27.0 GiB Logical volume root successfully resized
Finally, you'll have to resize the filesystem within /dev/turnkey/root so it can see that the underlying block device just got 10G bigger:
# resize2fs /dev/turnkey/root resize2fs 1.41.11 (14-Mar-2010) Filesystem at /dev/turnkey/root is mounted on /; on-line resizing required old desc_blocks = 2, new_desc_blocks = 2 Performing an on-line resize of /dev/turnkey/root to 7077888 (4k) blocks. The filesystem on /dev/turnkey/root is now 7077888 blocks long.
Have you tried using LVM? Still confused? Post a comment to discuss.
Comments
Is it better to mount an additional storage partition?
One of the first things I did with my TKL lamp server was increase the size of the root partition but in hindsight I've wondered if it would be better to leave it's size alone and mount an additional storage partition. Of course you can still use LVM to increase the second storage partition as needed.
It seems that would be a simpler method to manage storage and that method should work as well whether I am running the server in VMware or in Amazon EC2. It also seems like the separation of large data from the TKL appliance would inherently make it simpler to swap TKL appliances if needed.
More importantly, I'm wondering if there are any issues with the way Amazon instances and storage space are handled that would make either extending root partition or mounting an additional partition a more sensible approach for adding storage space. I haven't used Amazon's EC2 yet so I don't have any hands-on experience but my impression is that it would be simpler to keep the root partition as standard as possible and just mount additional storage partitions as needed.
Anyone have any insight they'd like to share?
Advanced configurations can be better, but are also more complex
If you install from the ISO the di-live installer allows you to setup a more advanced configuration if you want. TurnKey is Ubuntu under the hood so any partitioning scheme you would use for Ubuntu would also work with TurnKey. You may have to do some reconfiguring though (e.g., creating new logical volumes, moving data over, configuring mount points in fstab, etc.)
Regarding Amazon EC2, it lets you do exactly what you want via EBS (Elastic Block Store), which are high-speed SAN backed virtual storage devices that can be created and attached to an EC2 server instance on the fly. The Hub makes it very easy to use this functionality. You should try it.
Thanks. I agree that having
Thanks. I agree that having one partition for TurnKey makes the most sense as a default.
I'll definitely get around to trying EC2 so I can get a feel for things. I especially need to figure out how easy it is to add space to the EC2 instance's root partition.
Excellent blog post Liraz
I don't have immediate need for this info and have only just given it a quick scan (rather than a proper read) but appreciate it never the less. I will get back to this and have a good read as LVM is a bit of a mystery to me (I get the basic idea but it still seems a bit esoteric). I'm sure that many others will really appreciate it too.
Nice clarification
Thank you for the clarification. Much clearer now. Any potential of there being a webmin mod for this?
There already is...
There already is, and it's pre-installed in all TurnKey appliances v11+.
IIRC, log into webmin -> Hardware ->LVM
Fantastic!
Thanks so much for this Liraz! It worked a treat! Very easy to figure out due to your very simple and clear instructions =D
Lets elaborate on the IIRC Webmin Mod...
Last week I extended my Physical Volume in VMWare but ran into the block of not being able to expand the LVM. I missed the "middle part" about Volume Groups. I will study and Test. The Webmin module sounds good (for newbs like me).
Expanding the physical volume itself was pretty easy in VMware (use the Thin format), but unlike in windows, VMWare does not have an easy way to work with Linux Partitions after that - at least that I could find. The methodologies above will be more than helpful.
Partitioning in linux
Are you using TKL
TKL uses LVM by default, so this is very relevant for TKL users. If you are looking for more general Ubuntu partitioning info then you would probably want to have a look at this and this.
reboot not necessary !
Do you not have partprobe on your system?
Read man page.
Just what I needed...
This is great, thanks!
Loving TKL!
Jason
Just tested this myself and it worked like a charm!
Yay! :)
Whose idea was it to auto-subject? <.< >.>
I'm using tkFileServer and I used example #2.
I'm still pretty newb-ish with Linux, but, this helped me gain a better understanding of how LVM works.
I have 3 Windows 7 computers to back up: one gaming rig, one DAW, and a VMware server host; and 18~ GB is hardly enough space for one initial backup, much less three.
Thanks Liraz!
Transferring lvm volume to sda
Is there any way to transfer an lvm volume (ext4-formatted, i.e.) to a physical drive partition?
AFAIK the same way you'd transfer any partition to another...
So any sort of copying mechanism eg dd, rsync, etc. But probably easiest of all would be to do a TKLBAM backup of your appliance and do a clean install as you desire (no LVM if that's what you want) and restore the data in. Personally I'd test your backup before you destroy the current machine though, just in case something isn't quite right.
Out of interest why would you want to do that anyway? LVM takes a bit of getting used to but IMO it's way cool!
No luck with "dd", no need for "rsync", no need for TKLBAM, :-(
Thank you, Jeremy.
I was trying "dd" with not very much luck (you know, 20G could be very much data to do a dirty test). I don't have a TKLBAM account.
TK is a great appliance, but I found some compatibility issues with Eucalyptus. First (but not only) is the partitioning of the disk. Eucalyptus is expecting /dev/sda1 while TKL is giving /dev/mapper/turnkey-root as the root filesystem. In general, KVM support for TKL would be a great feature, I think.
Thank your very much again for your time.
Regards,
Roberto.
I have no experience with Eucalyptus
But I have heard some good things about it. I personally use ProxmoxVE and find it really good. TKL installs to KVM no worries (from ISO) and I have also converted lots of images to run under OVZ (PVE has both). Theoretically the TKL VM images should be able to be imported into KVM but I have only played with it once and dind't have any joy (I didn't try real hard because it's pretty quick and easy to install from ISO). But I agree a KVM image that could be easily imprted would be a plus.
dd is pretty slow but I have successfully dd'd up to 250GB images over a network (SHH) successfully on a number of occasions. That's how I migrated my physical Win2K3 server to a KVM VM. I just used a live CD and dd'd the HDD to a .raw image and KVM picked that up fine. But as I say that was with PVE not Eucalyptus so obviously YMMV.
We're going to publish images
We're going to publish images optimized for Eucalyptus soon. It would be great to get your feedback on that when it comes out. We'll post an update about it on the blog.
Eucalyptus issues and full support to KVM
Well, I'm not very experienced with Eucalyptus or KVM, but I've seen two many issues related to the "Community" version of Eucalyptus not seen in "Enterprise", too bad.
I'm very tempted to drop Eucalyptus in favor of any other virtualization platform (OpenStack, i.e.) or oVirt, when fully available. I don't use OpenVZ, so I see a great issue in using ISO instead of full installed templates like TKL, but it seems to be the only chance by now.
When I talk about KVM support, I am thinking of virtio drivers preinstalled and an XML file, what would be absolutelly amazing to spread TKL for LIBVIRT/KVM users.
ProxmoxVE seems impressive and very straightforward! It's great, but I'm thinking of an EC2 compatible API without loosing optimizations in every VM format. Eucalyptus is not the right option, since it seems centered in GNU/Linux images under Xen, no way of having Windows if you don't have Enterprise Edition.
Thanks.
Must admit that I've never installed drivers in TKL
I run TKL vanilla under KVM (on PVE) and my (anecdotal) experience has been that it seems close to bare metal performance - OVZ performance is even better (although perhaps with VirtIO drivers KVM would be equivalent?) Win is another story though! Without VirtIO drivers disk I/O can be noticably sluggish, especially under load. I haven't noticed much difference with the networking drivers though.
I understand the desire for EC2 compatability. I think OpenStack is probably the future in that regard but i guess time will tell. OTOH via the Hub using a TKLBAM backup I can migrate from KVM/OVZ/VirtualBox/bare metal to an AWS instance very easily (the Hub supports auto restore of a backup to EC2).
Anyway, good luck with your search and let us know if you find something good! :)
http://xenonhid4u.co.uk/
Thanks for this great and interesting article. I really enjoyed the article. It's really useful and informative for me.
LVM
Worked like a charm!
Thanks for the small bites of info for a first time LVM-er.
Thank you so much!
Only one suggestion: put a 'donate' link on this page so that I can buy you a beer. Thank you, worked perfectly!!
LVM (LOGICAL VOLUME MANAGER)
Looks good
Thanks for sharing....
You can get much more information from the below url,
http://www.digitalinux.com/2010/11/lvm-logical-volume-manager.html
It worked for me. Look like a
It worked for me. Look like a miracle! Thank so much.
pvcreate on unpartitioned hard disk
Hi! Thank you for the post! One question: Can I use pvcreate /dev/sdX on a unpartitioned hard disk or do I need to have some partition on it before running the command?
Thanks again!
You can use pvcreate on an unpartitioned disk
You can use pvcreate on an unpartitioned disk without any problem. In fact, that is what example #1 shows in the post. "pvcreate /dev/sdb"
Out of inodes
Hi
My problem is a little bit different - I am out of iNodes on my turnkey LVM...
What is my best cours of action ?
Any pointer appreciated
I just had a quick google
And from what I gather, it seems that the best course of action is to reformat and specify the number of inodes.
Have a read of these links for starters (although there may be betters one, they're just what came up via google):
http://serverfault.com/questions/111857/linux-help-im-running-out-of-inodes
http://www.linuxquestions.org/questions/linux-general-1/inode-lvm-and-in...
http://www.linuxquestions.org/questions/linux-newbie-8/running-out-of-in...
Top stuff but problematic?
Or is it just me ;)
I appreciate the conveniance of an LVM configuration but I've hit a few problems 'in production', normally in an abnormal power loss situation. As the OS detacts a 'IO failure' (reported by mount when attempting to resuce the drive), the machine doesn't boot. Someone needs to go in and reactive LVM and run a fsck before it'll boot. None of the LVM tools are available in a failed boot state so it becomes necessary to boot using another image or something.
I'm open to suggestions on a easy fix!
...fixed?
...been doing some research and I think a it just requires navigating through the grub rescue console as the LVM binaries are in the initrd image :) Haven't tested but a variation of the below:
And so on. Fingers crossed, I'll try that next time!
- spaceyjase
Post new comment