TurnKey Linux Virtual Appliance Library

Extending an LVM volume: Physical volumes (partitions) -> Volume groups -> Logical volume -> Filesystem

Logical Volume Management (AKA LVM) is a powerful, robust mechanism for managing storage space.

In TurnKey 11,  instead of installing the root filesystem directly to a fixed size partition, we setup LVM by default, and install the root filesystem to a Logical Volume, which may later be expanded, even across multiple physical devices.

Unfortunately, as with anything powerful, to get the most out of LVM you first have to negotiate a learning curve. From the feedback we've been getting it seems that confusion regarding LVM is  common with new users, so here's a quick "crash course"...

How LVM works

In LVM, there are several layers, each builds on top of the other:

PV[s] (Physical Volumes) -> VG[s] (Volume Groups) -> LV[s] (Logical Volumes) -> Filesystems.

Logical Volumes are allocated/extended within the boundaries of their underlying storage pool which is called a Volume Group in LVM terminology.

For example, in TurnKey the filesystem is installed by default to the /dev/turnkey/root Logical Volume, which is allocated within the turnkey Volume Group:

--- Logical volume ---
  LV Name                /dev/turnkey/root
  VG Name                turnkey
  LV UUID                OzX3fe-aRQa-81XM-0vCV-8aJo-eUL4-6J90XJ
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                17.0 GiB
  Current LE             4502
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:0

Out of the box the turnkey Volume Group doesn't have too much free space:

# vgdisplay
  --- Volume group ---
  VG Name               turnkey
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               18.14 GiB
  PE Size               4.00 MiB
  Total PE              4645
  Alloc PE / Size       4480 / 17.50 GiB
  Free  PE / Size       165 / 660.00 MiB
  VG UUID               IwaFL0-QCi8-iIUE-TWjQ-R906-PYpH-gMPaH9

We can only extend a Logical Volume within the free space of the underlying Volume Group. How much free space we currently have within the Volume Group can be seen in this part of the output:

Free  PE / Size       165 / 660.00 MiB

In the above example we only have 660 MB to allocate to LVMs within the turnkey Volume Group. So if we want to extend the root LV we'll have to first extend the VG backs it up.

Volume Groups group together Physical Volumes. That's why they're called Volume Groups. This command will show us which Physical Volumes have been registered into LVM, and to which volume groups they have been assigned:

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               turnkey
  PV Size               18.15 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              4645
  Free PE               165
  Allocated PE          4480
  PV UUID               H1Prpv-0VXR-7moE-zsbt-eyVt-m0fQ-GkAT6w

In this example we only have one Physical Volume (the /dev/sda2 partition) in the turnkey Volume Group.

Extending a Logical Volume

Bottom line: if the underlying Volume Group doesn't have enough free space, to extend the Logical Volume you'll first have to extend the underlying Volume Group by adding another Physical Volume to it.

In VMWare you could either create a new virtual hard disk device to add to the volume group, or extend an existing virtual hard disk device, create a new partition with cfdisk, and add the new partition to the Volume Group:

# example #1: you've added to VMWare a new virtual hard disk called /dev/sdb
pvcreate /dev/sdb
vgextend turnkey /dev/sdb

# example #2: you've expanded the existing sda hard disk
cfdisk /dev/sda  # creating /dev/sda3 (you may need to reboot before you can see this)
pvcreate /dev/sda3
vgextend turnkey /dev/sda3

After you've extended the Volume Group, you are free to extend the underlying Logical Volume:

# lvextend -L+10G /dev/turnkey/root
Extending logical volume root to 27.0 GiB
Logical volume root successfully resized

Finally, you'll have to resize the filesystem within /dev/turnkey/root so it can see that the underlying block device just got 10G bigger:

# resize2fs /dev/turnkey/root
resize2fs 1.41.11 (14-Mar-2010)
Filesystem at /dev/turnkey/root is mounted on /; on-line resizing required
old desc_blocks = 2, new_desc_blocks = 2
Performing an on-line resize of /dev/turnkey/root to  7077888 (4k) blocks.
The filesystem on /dev/turnkey/root is now 7077888 blocks long.

Have you tried using LVM? Still confused? Post a comment to discuss.

Enjoyed this post? Get future posts delivered by email or get the RSS feed.
Find us on Facebook and Twitter.
Last update: Tue, May 15 - 16:40

Comments

Is it better to mount an additional storage partition?

One of the first things I did with my TKL lamp server was increase the size of the root partition but in hindsight I've wondered if it would be better to leave it's size alone and mount an additional storage partition.  Of course you can still use LVM to increase the second storage partition as needed.

It seems that would be a simpler method to manage storage and that method should work as well whether I am running the server in VMware or in Amazon EC2.  It also seems like the separation of large data from the TKL appliance would inherently make it simpler to swap TKL appliances if needed.

More importantly, I'm wondering if there are any issues with the way Amazon instances and storage space are handled that would make either extending root partition or mounting an additional partition a more sensible approach for adding storage space.  I haven't used Amazon's EC2 yet so I don't have any hands-on experience but my impression is that it would be simpler to keep the root partition as standard as possible and just mount additional storage partitions as needed.

Anyone have any insight they'd like to share?

Liraz Siri's picture

Advanced configurations can be better, but are also more complex

In general, I think it's a good practice to partition the system stuff and big data separately, but it's also more complex to setup and maintain, which is why TurnKey defaults to one big root partition.

If you install from the ISO the di-live installer allows you to setup a more advanced configuration if you want. TurnKey is Ubuntu under the hood so any partitioning scheme you would use for Ubuntu would also work with TurnKey. You may have to do some reconfiguring though (e.g., creating new logical volumes, moving data over, configuring mount points in fstab, etc.)

Regarding Amazon EC2, it lets you do exactly what you want via EBS (Elastic Block Store), which are high-speed SAN backed virtual storage devices that can be created and attached to an EC2 server instance on the fly. The Hub makes it very easy to use this functionality. You should try it.

Thanks.  I agree that having

Thanks.  I agree that having one partition for TurnKey makes the most sense as a default.

I'll definitely get around to trying EC2 so I can get a feel for things.  I especially need to figure out how easy it is to add space to the EC2 instance's root partition.

Jeremy (aka JedMeister)'s picture

Excellent blog post Liraz

I don't have immediate need for this info and have only just given it a quick scan (rather than a proper read) but appreciate it never the less. I will get back to this and have a good read as LVM is a bit of a mystery to me (I get the basic idea but it still seems a bit esoteric). I'm sure that many others will really appreciate it too.


flexbean's picture

Nice clarification

Thank you for the clarification. Much clearer now. Any potential of there being a webmin mod for this?


Alon Swartz's picture

There already is...

There already is, and it's pre-installed in all TurnKey appliances v11+.

IIRC, log into webmin -> Hardware ->LVM

Gizmo's picture

Fantastic!

Thanks so much for this Liraz! It worked a treat! Very easy to figure out due to your very simple and clear instructions =D

L. Arnold's picture

Lets elaborate on the IIRC Webmin Mod...

Last week I extended my Physical Volume in VMWare but ran into the block of not being able to expand the LVM.  I missed the "middle part" about Volume Groups.   I will study and Test.  The Webmin module sounds good (for newbs like me).

Expanding the physical volume itself was pretty easy in VMware (use the Thin format), but unlike in windows, VMWare does not have an easy way to work with Linux Partitions after that - at least that I could find.  The methodologies above will be more than helpful.

Partitioning in linux

oh, Though i am a newbie for linux. I am using Ubuntu from last two years but in GUI mode. Abobve partitioning theory ,i found it more complex.can you help me how to partition in linux Ubuntu/Redhat using command line. Thanking you
Jeremy (aka JedMeister)'s picture

Are you using TKL

TKL uses LVM by default, so this is very relevant for TKL users. If you are looking for more general Ubuntu partitioning info then you would probably want to have a look at this and this.


reboot not necessary !

Do you not have partprobe on your system?

Read man page.

# example #2: you've expanded the existing sda hard disk
cfdisk /dev/sda  # creating /dev/sda3
partprobe
pvcreate /dev/sda3
vgextend turnkey /dev/sda3

Jason Lehman's picture

Just what I needed...

This is great, thanks!

Loving TKL!

Jason

Clay Fig's picture

Whose idea was it to auto-subject? <.< >.>

I'm using tkFileServer and I used example #2.

I'm still pretty newb-ish with Linux, but, this helped me gain a better understanding of how LVM works.

I have 3 Windows 7 computers to back up: one gaming rig, one DAW, and a VMware server host; and 18~ GB is hardly enough space for one initial backup, much less three.

Thanks Liraz!


Transferring lvm volume to sda

Is there any way to transfer an lvm volume (ext4-formatted, i.e.) to a physical drive partition?

Jeremy (aka JedMeister)'s picture

AFAIK the same way you'd transfer any partition to another...

So any sort of copying mechanism eg dd, rsync, etc. But probably easiest of all would be to do a TKLBAM backup of your appliance and do a clean install as you desire (no LVM if that's what you want) and restore the data in. Personally I'd test your backup before you destroy the current machine though, just in case something isn't quite right.

Out of interest why would you want to do that anyway? LVM takes a bit of getting used to but IMO it's way cool!


No luck with "dd", no need for "rsync", no need for TKLBAM, :-(

Thank you, Jeremy.

I was trying "dd" with not very much luck (you know, 20G could be very much data to do a dirty test). I don't have a TKLBAM account.

TK is a great appliance, but I found some compatibility issues with Eucalyptus. First (but not only) is the partitioning of the disk. Eucalyptus is expecting /dev/sda1 while TKL is giving /dev/mapper/turnkey-root as the root filesystem. In general, KVM support for TKL would be a great feature, I think.

Thank your very much again for your time.

Regards,

Roberto.

Jeremy (aka JedMeister)'s picture

I have no experience with Eucalyptus

But I have heard some good things about it. I personally use ProxmoxVE and find it really good. TKL installs to KVM no worries (from ISO) and I have also converted lots of images to run under OVZ (PVE has both). Theoretically the TKL VM images should be able to be imported into KVM but I have only played with it once and dind't have any joy (I didn't try real hard because it's pretty quick and easy to install from ISO). But I agree a KVM image that could be easily imprted would be a plus.

dd is pretty slow but I have successfully dd'd up to 250GB images over a network (SHH) successfully on a number of occasions. That's how I migrated my physical Win2K3 server to a KVM VM. I just used a live CD and dd'd the HDD to a .raw image and KVM picked that up fine. But as I say that was with PVE not Eucalyptus so obviously YMMV.


Liraz Siri's picture

We're going to publish images

We're going to publish images optimized for Eucalyptus soon. It would be great to get your feedback on that when it comes out. We'll post an update about it on the blog.

Eucalyptus issues and full support to KVM

Well, I'm not very experienced with Eucalyptus or KVM, but I've seen two many issues related to the "Community" version of Eucalyptus not seen in "Enterprise", too bad.

I'm very tempted to drop Eucalyptus in favor of any other virtualization platform (OpenStack, i.e.) or oVirt, when fully available. I don't use OpenVZ, so I see a great issue in using ISO instead of full installed templates like TKL, but it seems to be the only chance by now.

When I talk about KVM support, I am thinking of virtio drivers preinstalled and an XML file, what would be absolutelly amazing to spread TKL for LIBVIRT/KVM users.

ProxmoxVE seems impressive and very straightforward! It's great, but I'm thinking of an EC2 compatible API without loosing optimizations in every VM format. Eucalyptus is not the right option, since it seems centered in GNU/Linux images under Xen, no way of having Windows if you don't have Enterprise Edition.

Thanks.

Jeremy (aka JedMeister)'s picture

Must admit that I've never installed drivers in TKL

I run TKL vanilla under KVM (on PVE) and my (anecdotal) experience has been that it seems close to bare metal performance - OVZ performance is even better (although perhaps with VirtIO drivers KVM would be equivalent?) Win is another story though! Without VirtIO drivers disk I/O can be noticably sluggish, especially under load. I haven't noticed much difference with the networking drivers though.

I understand the desire for EC2 compatability. I think OpenStack is probably the future in that regard but i guess time will tell. OTOH via the Hub using a TKLBAM backup I can migrate from KVM/OVZ/VirtualBox/bare metal to an AWS instance very easily (the Hub supports auto restore of a backup to EC2).

Anyway, good luck with your search and let us know if you find something good! :)


http://xenonhid4u.co.uk/

Thanks for this great and interesting article. I really enjoyed the article. It's really useful and informative for me.

LVM

 

Thanks lol...
Below mentioned link is very easy to understand,
http://www.redhatlinux.info/2010/11/lvm-logical-volume-manager.html

Worked like a charm!

Thanks for the small bites of info for a first time LVM-er.

Thank you so much!

Only one suggestion: put a 'donate' link on this page so that I can buy you a beer. Thank you, worked perfectly!!

LVM (LOGICAL VOLUME MANAGER)

Looks good

Thanks for sharing....

You can get much more information from the below url,

http://www.digitalinux.com/2010/11/lvm-logical-volume-manager.html

It worked for me. Look like a

It worked for me. Look like a miracle! Thank so much.

pvcreate on unpartitioned hard disk

Hi! Thank you for the post! One question: Can I use pvcreate /dev/sdX on a unpartitioned hard disk or do I need to have some partition on it before running the command? 

Thanks again!

You can use pvcreate on an unpartitioned disk

You can use pvcreate on an unpartitioned disk without any problem.  In fact, that is what example #1 shows in the post.  "pvcreate /dev/sdb"

Out of inodes

Hi

My problem is a little bit different - I am out of iNodes on my turnkey LVM...

root@core /tmp# df -hi
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/turnkey-root
                        416K    416K     126  100% /
tmpfs                    63K       4     63K    1% /lib/init/rw
udev                     62K     514     62K    1% /dev
tmpfs                    63K       1     63K    1% /dev/shm
/dev/sda1               122K     207    122K    1% /boot
/dev/sda6               755K      18    755K    1% /storage

What is my best cours of action ?

Any pointer appreciated


Jeremy (aka JedMeister)'s picture

I just had a quick google

And from what I gather, it seems that the best course of action is to reformat and specify the number of inodes.

Have a read of these links for starters (although there may be betters one, they're just what came up via google):

http://serverfault.com/questions/111857/linux-help-im-running-out-of-inodes
http://www.linuxquestions.org/questions/linux-general-1/inode-lvm-and-in...
http://www.linuxquestions.org/questions/linux-newbie-8/running-out-of-in...


Top stuff but problematic?

Or is it just me ;)

I appreciate the conveniance of an LVM configuration but I've hit a few problems 'in production', normally in an abnormal power loss situation.  As the OS detacts a 'IO failure' (reported by mount when attempting to resuce the drive), the machine doesn't boot.  Someone needs to go in and reactive LVM and run a fsck before it'll boot.  None of the LVM tools are available in a failed boot state so it becomes necessary to boot using another image or something.

I'm open to suggestions on a easy fix!

...fixed?

...been doing some research and I think a it just requires navigating through the grub rescue console as the LVM binaries are in the initrd image :)  Haven't tested but a variation of the below:

  1. insmod ext2
  2. linux /vmlinuz-2.6.32-41-generic-pae
  3. initrd /inird.img-2.6.32-41-generic-pae
  4. /sbin/vgchange -a y
  5. fsck -y /dev/turnkey/root

And so on.  Fingers crossed, I'll try that next time!

 - spaceyjase

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)