Very Siberian's picture

Hi, sorry to be so desperate, but I tried following the instructions here to resize my LVM:

https://www.turnkeylinux.org/blog/extending-lvm

Now I cannot boot into TKL and I am stuck in Grub recovery mode. I have Googled all morning long without a fix. Any help would be greatly appreciated!

Thanks,

Rob

Forum: 
Very Siberian's picture

It ended up being faster just to redownload TKL and restore WordPress from a backup. However, I am left with the original problem of insufficient disk space, and I'm now terrified to follow the post I referenced above. Are there up to date instructions for how to safely resize the LVM?

Thanks,

Rob

Jeremy Davis's picture

When you followed that post, did you get any error messages? Was it all working ok (e.g. could you see the extra space etc) before you rebooted?

Also, out of interest, which path did you follow? That post contains 2 different pathways: Adding a new disk/volume (e.g. /dev/sdb) to the LVM. Or enlarging an existing disk/volume (by adding a new partition; /dev/sda3, to the LVM). I'm curious which path you attempted?

Regardless though, it still should have worked which ever path you used. We still install to LVM by default for the OVA and the instructions from that blog should still "just work" (I recently assisted another user using that post via our support portal). Even if it didn't work for some reason, I wouldn't have expected you to hit a grub prompt on reboot?! The boot shouldn't be affected. And unfortunately, without seeing some output from the grub prompt (e.g. 'ls (hd0, 1)'), I'd only be guessing what might have happened.

Regardless, there are 2 ways forward that I can think of.

One would be to do the resizing locally (before uploading an OVA). IIRC the OVA should use a "sparse" disk. So even after resizing, it shouldn't actually physically get too much bigger (it might be a little bigger, but not as much as the full free size). Before you upload, test a reboot first. You may still hit the same issue, but at least you can play with it locally, with no pressure.

The other way, would be to add a new volume to the existing server, but instead of adding it to the LVM, just mount bind it to where you want the space (e.g. /var/www/wordpress). I'm not super familiar with how to add a new volume on Google Cloud, but once you've added it and assuming it's /dev/sdb (it's likely not) then it would go something like this (please test this first; it's off the top of my head and there is a chance I've missed something or remembered it wrong):

fdisk /dev/sdb # if you select the wrong one you might destroy your OS
# this will run interactively
# create a new partition using 'n' and select defaults hitting enter
# save and exit with 'w'
mkfs.ext4 /dev/sdb1
mkdir /media/extra-volume
mount /dev/sdb1 /media/extra-volume
mv /var/www/wordpress /media/extra-volume/
mkdir /var/www/wordpress
mount --bind /media/extra-volume/wordpress /var/www/wordpress
# assuming you want it all owned by the webserver (www-data)
chown -R www-data:www-data /var/www/wordpress

To ensure that it survives reboot, you'll need to edit the fstab:

cat >> /etc/fstab <<EOF
/dev/sdb1 /media/extra-volume ext4 errors=remount-ro 0 1
/media/extra-volume/wordpress /var/www/wordpress none bind 0 0
EOF

Then to double check that your fstab updates work, unmount it all:

umount /var/www/wordpress
umount /media/extra-volume
mount -a

If that gives no error messages, then check to see that they're remounted and show up with this command:

mount | grep /dev/sdb
Very Siberian's picture

Thanks for your reply. Here are my replies.

When you followed that post, did you get any error messages? Was it all working ok (e.g. could you see the extra space etc) before you rebooted?

I did not get any error messages and was able to see the extra space before I rebooted.

Also, out of interest, which path did you follow? That post contains 2 different pathways: Adding a new disk/volume (e.g. /dev/sdb) to the LVM. Or enlarging an existing disk/volume (by adding a new partition; /dev/sda3, to the LVM). I'm curious which path you attempted?

I used the second pathway: add a new partition from the free space to the LVM.

The instance to which I'm referring is not one in Google Cloud (but thanks for remembering my other thread!). It is a locally hosted VM instance in VirtualBox. Incidentally, even after redoing the VM completely and installing the WordPress site from backup, I'm in the same position I was in originally of needing to increase disk space. I am a bit terrified of that blog post given what happened but need to try again. Please let me know if you have any suggestions. Here is where I'm starting from now:

1. Volume Group: turnkey (18.57 GiB)

2. Physical Volume: sda1 (18.57 GiB)

3. Logical Volumes: root (17 GiB) and swap_1 (512 MB)

I increased the virtual disk image housing the appliance to 100 GB. I just need to safely allow the LVM to capture the additional disk space. When I run cfdisk, I see the following:

- Free space (59.1 M)

- /dev/sda1 (boot) 18.6G Type Linux LVM

- Free space (81.4G)

Best regards,

Rob

Jeremy Davis's picture

I downloaded the current v16.1 WordPress OVA (turnkey-wordpress-16.1-buster-amd64.ova). Unfortunately I didn't initially allow VirtualBox to import the virtual disk as it's native VDI format (instead left it as VMDK). And as VirtualBox can't resize VMDKs, I had to convert it to VDI first. Assuming that you did allow it to import as VDI (the default) you wouldn't have needed to do that. I stopped the VM and detached the vmdk file. While I was at it, I also resized it from the commandline:

# from directory where the VM files are
# convert vmdk to vdi
VBoxManage clonehd turnkey-wordpress-16.1-buster-amd64-disk1.vmdk wordpress-16.1.vdi --format vdi
# resize vdi to 30GB
VBoxManage modifyhd wordpress-16.1.vdi --resize 30720

Once I'd done that, I attached the new vdi file to my VM and booted it. So far, so good... I'm assuming that this is essentially where you are starting from. From here on, I logged in via SSH (IMO working in the VM window is a PITA).

I tried following in your footsteps, but I've been unable to reproduce your issue. I've given verbose output of what I did though, so perhaps you may have done something different?

When I look at the output of cfdisk, I'm immediately struck by the fact that there is ~59MB partition at the start. You need to be sure NOT to change that. I wonder if that may be your issue?! Just partition the 2nd block of free space (obviously your free space will be bigger):

                                Disk: /dev/sda
               Size: 30 GiB, 32212254720 bytes, 62914560 sectors
                      Label: dos, identifier: 0x89d293bd

    Device       Boot        Start        End    Sectors    Size  Id Type
    Free space                2048     123045     120998   59.1M
    /dev/sda1    *          123046   39062500   38939455   18.6G  8e Linux LVM
>>  Free space            39063552   62914559   23851008   11.4G               

Then create the partition. You could change the "type" to '8e' ('Linux LVM' - it will default to '83' - 'Linux') if you want, but I left it as is:

                                Disk: /dev/sda
               Size: 30 GiB, 32212254720 bytes, 62914560 sectors
                      Label: dos, identifier: 0x89d293bd

    Device       Boot        Start        End    Sectors    Size  Id Type
    Free space                2048     123045     120998   59.1M
    /dev/sda1    *          123046   39062500   38939455   18.6G  8e Linux LVM
>>  /dev/sda2             39063552   62914559   23851008   11.4G  83 Linux     
 ┌───────────────────────────────────────────────────────────────────────────┐
 │Partition type: Linux (83)                                                 │
 └───────────────────────────────────────────────────────────────────────────┘
    [Bootable]  [ Delete ]  [ Resize ]  [  Quit  ]  [  Type  ]  [  Help  ]
    [  Write ]  [  Dump  ]

            Write partition table to disk (this might destroy data)

Select "Write" and follow the prompts. For good measure, I double checked via fdisk:

root@wordpress ~# fdisk -l /dev/sda
Disk /dev/sda: 30 GiB, 32212254720 bytes, 62914560 sectors
Disk model: HARDDISK        
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x89d293bd

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda1  *      123046 39062500 38939455 18.6G 8e Linux LVM
/dev/sda2       39063552 62914559 23851008 11.4G 83 Linux

Then I just followed the blog post:

root@wordpress ~# pvcreate /dev/sda2
  Physical volume "/dev/sda2" successfully created.
root@wordpress ~# vgextend turnkey /dev/sda2
  Volume group "turnkey" successfully extended
root@wordpress ~# lvextend -L+10G /dev/turnkey/root
  Size of logical volume turnkey/root changed from 17.00 GiB (4352 extents) to 27.00 GiB (6912 extents).
  Logical volume turnkey/root successfully resized.
root@wordpress ~# resize2fs /dev/turnkey/root
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/turnkey/root is mounted on /; on-line resizing required
old_desc_blocks = 3, new_desc_blocks = 4
The filesystem on /dev/turnkey/root is now 7077888 (4k) blocks long.

Note that most Linux utilities use "GiB" (Gibibytes - i.e. 1024 x 1024 x 1024 bytes; as opposed to GB/Gigabytes - 1000 x 1000 x 1000 bytes). Because your vHDD is 100GB, when you run the lvextend command, you'll probably want to '+80G'. I.e.:

lvextend -L+80G /dev/turnkey/root

Alternatively, if you just want to use all of the free space. Do that like this:

lvextend -l+100%FREE /dev/turnkey/root

Note that when using percentage like that, you need to use lower case L.

Finally I double checked that the new space is visible:

root@wordpress ~# df -h /
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/turnkey-root   27G  1.4G   24G   6% /

Then I rebooted:

reboot && exit

The reboot worked as I would have expected. And other than that small free space at the start of the disk, I'm not really sure why it didn't work for you?! The only other thing that occurs to me is do you definitely have enough free space on the host? TBH, I wouldn't expect that to cause the issue you're hitting, but perhaps?

Very Siberian's picture

Thank you, Jeremy! This worked perfectly. With the benefit of not being under duress and your expanded instructions, everything went smoothly. In hindsight, I believe that I erred in selecting an extended (rather than primary) partition for sda2. I probably also selected the wrong filesystem since I don't recall it being a Linux filesystem. I'm still not sure why GRUB got hosed and rendered my whole appliance non-bootable, but in any case, I acknowledge that it was almost certainly user error and not something faulty in the blog post. Thank you again for your detailed reply. I now have exactly what I needed: a much larger VDI that the OS recognizes.

Best regards,

Rob

Jeremy Davis's picture

Glad to hear that you are up and running - with extra space! :)

It is an interesting issue though. For what it's worth cfdisk should really be asking whether you want to create a "primary logical" partition, an "extended" partition (which is a primary partition) or a "secondary logical" partition (inside an extended partition if one already exists). That is because an extended partition is simply a container for additional (aka secondary) logical partitions to reside in.

Extended partitions are a hack to work around the limitations of the old partitioning format provided by MBR (Master Boot Record). MBR can only support 4 (primary) partitions. The newer GPT (GUID Partition Table) doesn't have those same limitations, but generally requires support for UEFI boot (which older hardware and default VM config doesn't provide). We don't currently support UEFI boot (but hope to in the future - only for ISO installs though).

AFAIK only logical partitions are usable for filesystems. By my understanding an extended partition isn't directly usable. It's value is only as a container for additional logical partitions, and only the logical partitions contained within it can be formatted. But (assuming that I understand correctly) it seems that LVM allowed you to add an extended partition as a PV (physical volume) and extend the filesystem into the "free" space (that wasn't actually usable). So whilst it appeared to work, it actually didn't and that only became apparent on reboot. If that was indeed the case, it's probably a really good thing that it failed to boot; otherwise it might have just randomly lost data...!

When I get a chance, I'll see if I can recreate the issue and if I can, I'll report the bug to LVM maintainers upstream.

Add new comment