TurnKey Linux Virtual Appliance Library

Extending an LVM volume: Physical volumes (partitions) -> Volume groups -> Logical volume -> Filesystem

Logical Volume Management (AKA LVM) is a powerful, robust mechanism for managing storage space.

In TurnKey 11,  instead of installing the root filesystem directly to a fixed size partition, we setup LVM by default, and install the root filesystem to a Logical Volume, which may later be expanded, even across multiple physical devices.

Unfortunately, as with anything powerful, to get the most out of LVM you first have to negotiate a learning curve. From the feedback we've been getting it seems that confusion regarding LVM is  common with new users, so here's a quick "crash course"...

How LVM works

In LVM, there are several layers, each builds on top of the other:

PV[s] (Physical Volumes) -> VG[s] (Volume Groups) -> LV[s] (Logical Volumes) -> Filesystems.

Logical Volumes are allocated/extended within the boundaries of their underlying storage pool which is called a Volume Group in LVM terminology.

For example, in TurnKey the filesystem is installed by default to the /dev/turnkey/root Logical Volume, which is allocated within the turnkey Volume Group:

--- Logical volume ---
  LV Name                /dev/turnkey/root
  VG Name                turnkey
  LV UUID                OzX3fe-aRQa-81XM-0vCV-8aJo-eUL4-6J90XJ
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                17.0 GiB
  Current LE             4502
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:0

Out of the box the turnkey Volume Group doesn't have too much free space:

# vgdisplay
  --- Volume group ---
  VG Name               turnkey
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               18.14 GiB
  PE Size               4.00 MiB
  Total PE              4645
  Alloc PE / Size       4480 / 17.50 GiB
  Free  PE / Size       165 / 660.00 MiB
  VG UUID               IwaFL0-QCi8-iIUE-TWjQ-R906-PYpH-gMPaH9

We can only extend a Logical Volume within the free space of the underlying Volume Group. How much free space we currently have within the Volume Group can be seen in this part of the output:

Free  PE / Size       165 / 660.00 MiB

In the above example we only have 660 MB to allocate to LVMs within the turnkey Volume Group. So if we want to extend the root LV we'll have to first extend the VG backs it up.

Volume Groups group together Physical Volumes. That's why they're called Volume Groups. This command will show us which Physical Volumes have been registered into LVM, and to which volume groups they have been assigned:

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               turnkey
  PV Size               18.15 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              4645
  Free PE               165
  Allocated PE          4480
  PV UUID               H1Prpv-0VXR-7moE-zsbt-eyVt-m0fQ-GkAT6w

In this example we only have one Physical Volume (the /dev/sda2 partition) in the turnkey Volume Group.

Extending a Logical Volume

Bottom line: if the underlying Volume Group doesn't have enough free space, to extend the Logical Volume you'll first have to extend the underlying Volume Group by adding another Physical Volume to it.

In VMWare you could either create a new virtual hard disk device to add to the volume group, or extend an existing virtual hard disk device, create a new partition with cfdisk, and add the new partition to the Volume Group:

# example #1: you've added to VMWare a new virtual hard disk called /dev/sdb
pvcreate /dev/sdb
vgextend turnkey /dev/sdb

# example #2: you've expanded the existing sda hard disk
cfdisk /dev/sda  # creating /dev/sda3 (you may need to reboot before you can see this)
pvcreate /dev/sda3
vgextend turnkey /dev/sda3

After you've extended the Volume Group, you are free to extend the underlying Logical Volume:

# lvextend -L+10G /dev/turnkey/root
Extending logical volume root to 27.0 GiB
Logical volume root successfully resized

Finally, you'll have to resize the filesystem within /dev/turnkey/root so it can see that the underlying block device just got 10G bigger:

# resize2fs /dev/turnkey/root
resize2fs 1.41.11 (14-Mar-2010)
Filesystem at /dev/turnkey/root is mounted on /; on-line resizing required
old desc_blocks = 2, new_desc_blocks = 2
Performing an on-line resize of /dev/turnkey/root to  7077888 (4k) blocks.
The filesystem on /dev/turnkey/root is now 7077888 blocks long.

Have you tried using LVM? Still confused? Post a comment to discuss.

You can get future posts delivered by email or good old-fashioned RSS.
TurnKey also has a presence on Google+, Twitter and Facebook.

Comments

Liraz Siri's picture

Advanced configurations can be better, but are also more complex

In general, I think it's a good practice to partition the system stuff and big data separately, but it's also more complex to setup and maintain, which is why TurnKey defaults to one big root partition.

If you install from the ISO the di-live installer allows you to setup a more advanced configuration if you want. TurnKey is Ubuntu under the hood so any partitioning scheme you would use for Ubuntu would also work with TurnKey. You may have to do some reconfiguring though (e.g., creating new logical volumes, moving data over, configuring mount points in fstab, etc.)

Regarding Amazon EC2, it lets you do exactly what you want via EBS (Elastic Block Store), which are high-speed SAN backed virtual storage devices that can be created and attached to an EC2 server instance on the fly. The Hub makes it very easy to use this functionality. You should try it.

Jeremy's picture

Excellent blog post Liraz

I don't have immediate need for this info and have only just given it a quick scan (rather than a proper read) but appreciate it never the less. I will get back to this and have a good read as LVM is a bit of a mystery to me (I get the basic idea but it still seems a bit esoteric). I'm sure that many others will really appreciate it too.

flexbean's picture

Nice clarification

Thank you for the clarification. Much clearer now. Any potential of there being a webmin mod for this?


Alon Swartz's picture

There already is...

There already is, and it's pre-installed in all TurnKey appliances v11+.

IIRC, log into webmin -> Hardware ->LVM

Gizmo's picture

Fantastic!

Thanks so much for this Liraz! It worked a treat! Very easy to figure out due to your very simple and clear instructions =D

L. Arnold's picture

Lets elaborate on the IIRC Webmin Mod...

Last week I extended my Physical Volume in VMWare but ran into the block of not being able to expand the LVM.  I missed the "middle part" about Volume Groups.   I will study and Test.  The Webmin module sounds good (for newbs like me).

Expanding the physical volume itself was pretty easy in VMware (use the Thin format), but unlike in windows, VMWare does not have an easy way to work with Linux Partitions after that - at least that I could find.  The methodologies above will be more than helpful.

Partitioning in linux

oh, Though i am a newbie for linux. I am using Ubuntu from last two years but in GUI mode. Abobve partitioning theory ,i found it more complex.can you help me how to partition in linux Ubuntu/Redhat using command line. Thanking you
Jeremy's picture

Are you using TKL

TKL uses LVM by default, so this is very relevant for TKL users. If you are looking for more general Ubuntu partitioning info then you would probably want to have a look at this and this.

reboot not necessary !

Do you not have partprobe on your system?

Read man page.

# example #2: you've expanded the existing sda hard disk
cfdisk /dev/sda  # creating /dev/sda3
partprobe
pvcreate /dev/sda3
vgextend turnkey /dev/sda3

You need parted istalled,

You need parted istalled, which, for some reason, is not included in the core turnkey setup.

Jason Lehman's picture

Just what I needed...

This is great, thanks!

Loving TKL!

Jason

Jeremy's picture

Just tested this myself and it worked like a charm!

Yay! :)

Transferring lvm volume to sda

Is there any way to transfer an lvm volume (ext4-formatted, i.e.) to a physical drive partition?

Jeremy's picture

AFAIK the same way you'd transfer any partition to another...

So any sort of copying mechanism eg dd, rsync, etc. But probably easiest of all would be to do a TKLBAM backup of your appliance and do a clean install as you desire (no LVM if that's what you want) and restore the data in. Personally I'd test your backup before you destroy the current machine though, just in case something isn't quite right.

Out of interest why would you want to do that anyway? LVM takes a bit of getting used to but IMO it's way cool!

No luck with "dd", no need for "rsync", no need for TKLBAM, :-(

Thank you, Jeremy.

I was trying "dd" with not very much luck (you know, 20G could be very much data to do a dirty test). I don't have a TKLBAM account.

TK is a great appliance, but I found some compatibility issues with Eucalyptus. First (but not only) is the partitioning of the disk. Eucalyptus is expecting /dev/sda1 while TKL is giving /dev/mapper/turnkey-root as the root filesystem. In general, KVM support for TKL would be a great feature, I think.

Thank your very much again for your time.

Regards,

Roberto.

Jeremy's picture

I have no experience with Eucalyptus

But I have heard some good things about it. I personally use ProxmoxVE and find it really good. TKL installs to KVM no worries (from ISO) and I have also converted lots of images to run under OVZ (PVE has both). Theoretically the TKL VM images should be able to be imported into KVM but I have only played with it once and dind't have any joy (I didn't try real hard because it's pretty quick and easy to install from ISO). But I agree a KVM image that could be easily imprted would be a plus.

dd is pretty slow but I have successfully dd'd up to 250GB images over a network (SHH) successfully on a number of occasions. That's how I migrated my physical Win2K3 server to a KVM VM. I just used a live CD and dd'd the HDD to a .raw image and KVM picked that up fine. But as I say that was with PVE not Eucalyptus so obviously YMMV.

Liraz Siri's picture

We're going to publish images

We're going to publish images optimized for Eucalyptus soon. It would be great to get your feedback on that when it comes out. We'll post an update about it on the blog.

Eucalyptus issues and full support to KVM

Well, I'm not very experienced with Eucalyptus or KVM, but I've seen two many issues related to the "Community" version of Eucalyptus not seen in "Enterprise", too bad.

I'm very tempted to drop Eucalyptus in favor of any other virtualization platform (OpenStack, i.e.) or oVirt, when fully available. I don't use OpenVZ, so I see a great issue in using ISO instead of full installed templates like TKL, but it seems to be the only chance by now.

When I talk about KVM support, I am thinking of virtio drivers preinstalled and an XML file, what would be absolutelly amazing to spread TKL for LIBVIRT/KVM users.

ProxmoxVE seems impressive and very straightforward! It's great, but I'm thinking of an EC2 compatible API without loosing optimizations in every VM format. Eucalyptus is not the right option, since it seems centered in GNU/Linux images under Xen, no way of having Windows if you don't have Enterprise Edition.

Thanks.

Jeremy's picture

Must admit that I've never installed drivers in TKL

I run TKL vanilla under KVM (on PVE) and my (anecdotal) experience has been that it seems close to bare metal performance - OVZ performance is even better (although perhaps with VirtIO drivers KVM would be equivalent?) Win is another story though! Without VirtIO drivers disk I/O can be noticably sluggish, especially under load. I haven't noticed much difference with the networking drivers though.

I understand the desire for EC2 compatability. I think OpenStack is probably the future in that regard but i guess time will tell. OTOH via the Hub using a TKLBAM backup I can migrate from KVM/OVZ/VirtualBox/bare metal to an AWS instance very easily (the Hub supports auto restore of a backup to EC2).

Anyway, good luck with your search and let us know if you find something good! :)

http://xenonhid4u.co.uk/

Thanks for this great and interesting article. I really enjoyed the article. It's really useful and informative for me.

LVM

 

Thanks lol...
Below mentioned link is very easy to understand,
http://www.redhatlinux.info/2010/11/lvm-logical-volume-manager.html

Worked like a charm!

Thanks for the small bites of info for a first time LVM-er.

Thank you so much!

Only one suggestion: put a 'donate' link on this page so that I can buy you a beer. Thank you, worked perfectly!!

LVM (LOGICAL VOLUME MANAGER)

Looks good

Thanks for sharing....

You can get much more information from the below url,

http://www.digitalinux.com/2010/11/lvm-logical-volume-manager.html

It worked for me. Look like a

It worked for me. Look like a miracle! Thank so much.

pvcreate on unpartitioned hard disk

Hi! Thank you for the post! One question: Can I use pvcreate /dev/sdX on a unpartitioned hard disk or do I need to have some partition on it before running the command? 

Thanks again!

You can use pvcreate on an unpartitioned disk

You can use pvcreate on an unpartitioned disk without any problem.  In fact, that is what example #1 shows in the post.  "pvcreate /dev/sdb"

Out of inodes

Hi

My problem is a little bit different - I am out of iNodes on my turnkey LVM...

root@core /tmp# df -hi
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/turnkey-root
                        416K    416K     126  100% /
tmpfs                    63K       4     63K    1% /lib/init/rw
udev                     62K     514     62K    1% /dev
tmpfs                    63K       1     63K    1% /dev/shm
/dev/sda1               122K     207    122K    1% /boot
/dev/sda6               755K      18    755K    1% /storage

What is my best cours of action ?

Any pointer appreciated


Jeremy's picture

I just had a quick google

And from what I gather, it seems that the best course of action is to reformat and specify the number of inodes.

Have a read of these links for starters (although there may be betters one, they're just what came up via google):

http://serverfault.com/questions/111857/linux-help-im-running-out-of-inodes
http://www.linuxquestions.org/questions/linux-general-1/inode-lvm-and-in...
http://www.linuxquestions.org/questions/linux-newbie-8/running-out-of-in...

Top stuff but problematic?

Or is it just me ;)

I appreciate the conveniance of an LVM configuration but I've hit a few problems 'in production', normally in an abnormal power loss situation.  As the OS detacts a 'IO failure' (reported by mount when attempting to resuce the drive), the machine doesn't boot.  Someone needs to go in and reactive LVM and run a fsck before it'll boot.  None of the LVM tools are available in a failed boot state so it becomes necessary to boot using another image or something.

I'm open to suggestions on a easy fix!

...fixed?

...been doing some research and I think a it just requires navigating through the grub rescue console as the LVM binaries are in the initrd image :)  Haven't tested but a variation of the below:

  1. insmod ext2
  2. linux /vmlinuz-2.6.32-41-generic-pae
  3. initrd /inird.img-2.6.32-41-generic-pae
  4. /sbin/vgchange -a y
  5. fsck -y /dev/turnkey/root

And so on.  Fingers crossed, I'll try that next time!

 - spaceyjase

Thanks

It was very easy to follow your steps , Thanks  a Lot

thanks !

this (and the fact you decided for LVM) literally saved my life when i created a VirtualBox fixed drive and it showed too small later ...

i tried adding the -r option to lvextend. it came with some interesting warning about damaging my system, so i said no, but resize2fs was ran for me anyway :)

Many thanks!!!

Great step-by-step tutorial.

Many thanks.

regarding lvextend

lvextend -l +2047 /dev/vgone/lv_usetest

 

What does +2047 stand for i know it is for 8gb how do we calculate.

lvm + encryption

Hello,

 

Is there any way to support lvm+encryption for truecrypt installation? This could be particularly usefull when administrator of different vms have access to the host (the hypervisor) in order to backup, restart (whatever) their own virtual appliance. If each vm is encrypted, an admin of a virtual will not be able to read and understand the content of another.

Jeremy's picture

Is definitely possible

But it is not supportted OOTB so you'd need to configure this manually on each guest (although perhaps you could script it to somewhat automate the process).

My suggestion would be to have a look at ProxmoxVE. It allows the configuring of users and groups which each can have certain levels of control/access on a per vm, or per host/node basis. It also allows TKL (and other Linux distros) to run under OVZ (which gives you 97-99% performance of bare metal) or KVM (which supports Windows and other OS).

Great post!

Great post. Worked like a charm for me. I used to increase lvm partition on a Red Hat Virtualization VM. 

Thank you.

Great Tutorial

This made it a lot easier on me to extend my storage.

Superb post

This just saved my bacon somewhat - thanks for an excellent post.

Hero

Thank you so much this was exactly the problem i had. solved!

Can't expand LVM to make a filesystem

Hi

I might have misunderstood something here, but I will try and ask my question anyway, as it seems to be a great place for knowlegde about LVM.

 

Long story short:

I messed up an upgrade of my ReadyNAS Duo from netgear. It runs on linux and stupid me made a normal apt-get upgrade as I often do on my Ubuntu.

After the upgrade I couldn't get in contact with my serve/disk (with all my pics, music and movies). I took the disk and put it into an usb-dockingstation to see what was wrong.

The disk (1TB) should at least have 250 GB free but I get something completly different when I test it with different commands

 

root@martin-ubudesktop:/home/martin# lvdisplay /dev/c/c
  --- Logical volume ---
  LV Path                /dev/c/c
  LV Name                c
  VG Name                c
  LV UUID                0h6FoC-k5xk-Ziax-d06Z-V8lW-AcOK-A0haBA
  LV Write Access        read/write
  LV Creation host, time , 
  LV Status              available
  # open                 0
  LV Size                929,09 GiB
  Current LE             29731
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:0
   
root@martin-ubudesktop:/home/martin# vgdisplay c
  --- Volume group ---
  VG Name               c
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               929,09 GiB
  PE Size               32,00 MiB
  Total PE              29731
  Alloc PE / Size       29731 / 929,09 GiB
  Free  PE / Size       0 / 0   
  VG UUID               8mo4RP-AHcs-Hr3r-yQ1Z-uWYw-WaUo-dEEYkJ
 
I think the LV and LG names came form the ReadyNAS.
 
As I have understood this guide, I need some space to make some kind of filesystem (so I can access my files?)
If my VG doesn't have enough free space, can I then (to extend VL) use my desktop's harddisk to create a PV  and then add it to the VG? (example #1???)
 
I know am on deep water here and I'm soon to go down for the third time.....and my wife doesn't know about it yet!
 
Best regards and thanks for any kind of help
Martin :-)

  

P.S.

Can I by the way use gparted to shrink the LVM partion?

As I said there should at least still be 250 GB free on the disk.

Possible solution

You might want to take a look at this:

http://www.readynas.com/forum/viewtopic.php?f=11&t=35153

I've recovered 100% of files from a couple of borked readynas systems using that methodology. 

Hope it helps!

The link you gave me looked

The link you gave me looked really promising, and while I was downloading the Ubuntu image I scrolled through the thread finding this post (page 9 third from the bottom)
 

Re: ReadyNAS Data Recovery - VMware recovery tool

Postby doyley86 » Fri Dec 28, 2012 1:19 pm

Guys I have been struggling with this for months, I have found a simple easier way of restoring all my files using some free software called DiskInternals Linux Reader you can download it free from here:http://www.diskinternals.com/linux-reader/ 

it finds your drive straight away and allows you to copy off all your data! Hope this helps.

Cheers
 
It's just a small windows program but it reads my LVM without any problems, and now I'm backing everything up nice and easy.
I will save the VMware trick for another catastrophe - for I'm pretty sure there will be more like this one to come.
Oh, and by the way. My wife has forgiven me now!!!
 
Martin :-)
Jeremy's picture

Excuse me if I missed something...

But why not just mount your LV and then copy out the files?

Something like this should work:

mount /dev/c/c /mnt

Then all your files should be somewhere in /mnt (I have no idea what subfolder - it depends on where ReadyNAS stores stuff).

You didn't miss anything

It's me who might not have been given all relavant info about my drive.

I have tried to mount the drive in any possible way.

Often I got a mesage about bad superblocks and I would try and restore the superblock with the backup superblocks, but that never worked.

Sometimes I got other messages (something about zero-lenght partitions, invalid filesystem and so on) and I could nevet get past those.

Maybe if I have had somebody who would know a lot more about this than me (and that shouldn't be hard to find), but I really tried A LOT of things.

Then I came to this page and the rest is history...

 

Martin :-)

Jeremy's picture

Ahh ok

That's a whole other issue then. My advice to start with would be to take an image of the drive before doing anything. Ideally I like to image to a same or larger size drive. dd is the go, or ddrescue if there are hardware or data corruption issues.

Otherwise I'm not much help. I had a HDD fail on me some time ago and I managed to recover some of the data (about a quarter) by writing a new super block, but I don't recall how I did it... Sorry...

Glad it got you on the right

Glad it got you on the right track!

Nice article and lvextend options

Nice article -

Two helpful things I want to add which I found out during my lvm "studies" as well

The PE's are really nice (from vgdisplay command) and give you some very fine tuned control if you want it.  For example if you want to make sure to use all of the free space available, specifying a number such as "1G" or "10G" may not do it.

From one of the vgdisplay command you have

VG Size               18.14 GiB
  PE Size               4.00 MiB
  Total PE              4645
  Alloc PE / Size       4480 / 17.50 GiB
  Free  PE / Size       165 / 660.00 MiB

The PE size is 4mb, and there are 165 free PEs (660mb)

For lvextend you can use

lvextend -l +165 [lvname]

With no units by default it's the # PE

Another helpful shortcut to use all space available

lvextend -l +100%FREE

Thanks!

p.s.http://geekswing.com/geek/unix/extending-a-linux-disk-with-lvm-extending-root-partition/

Each disk for each user

How i CAN USe each partition for each user turnkey file server ? or better say each disk for example 1 tera byte for each user? thanks

Jeremy's picture

I have no idea

But I'm sure if you search on google you'll find plenty of info. Keep in mind that TKL v12.x is based on Debian Squeeze and I'm sure you'll find something. Hint: I think you are after user quotas using LVM

Each disk for each user

Thanks Jeremy, I will search on google about that information. My plan is use around 8 disk (usb disk) of 1 tera, and then use that 8  disk connect in RAID mode,  mode mirror (2 disk mirrored=1 tera for 4 users), and I think turnkey is very powerfull tool for that aplication.

muy question is, you can do this with turnkey? (http://upload.wikimedia.org/wikipedia/commons/c/ce/RAID_100.svg), and then make quotas using LVM?

thanks a lot

Brian

Jeremy's picture

AFAIK theoretically

But I've never done it... I think mdam is the one to use for software RAID.

The only thing is that I'm not sure if LVM would really be relevant then... AFAIK the way that quotas work is that they use separate partitions...

LVM usage

I tried using the webmin lvm tool, but then I read your post.  Your explanation was the best I've seen yet.  I followed and everything worked like a charm.  I did have to reboot as you said might happen.  Also note, 1.  issuing resize2fs may take several minutes 2. for me webmin disk usage side bar shows the original free space now consumed - odd but no big deal.

 

Thanks for the post. 

Thank You!!!

Thank you so much, man. I've been banging my head against the wall trying to figure this our and this resolved it perfectly within 5 minutes.

LVM IT AWESOME REAL

This kind of the file system is very amaizing and from now on  will be using this kind of file system into my system.....

Thank you

This was exactly what I needed. Thank you for the detailed post!

What I needed after 2 1/2 days of frustration

I wasn't trying to do anything with Turnkey but rather was trying to extend a VirtualBox partition. The information on LVM was just what was needed for this Linux newbie. Thank you!

Thank you

Thank you for this post!

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)