Bryan's picture
Bryan - Mon, 2015/08/17 - 19:53

I have a problem I cannot seem to solve. A few days ago my mediawiki appliance, which has been running without flaw for >4 years, stopped working. When I try to access it I get the error:

Sorry! This site is experiencing technical difficulties.
Try waiting a few minutes and reloading.

(Can't contact the database server: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) (localhost))

If I go into webadmin the MySQL server is not running; if I click the 'start mysql" button I get the following error:

MySQL is not running on your system - database list could not be retrieved.

Sometimes it also states that it could not create a temporary folder; after a bit of fiddling it turns out this was due to the virtual hard drive bing 100% full, which was unexpected as when I set up the wiki origonally its drive was 3X the installed size, the wiki is quite small, and there are no images on it.

So I went through an extensive cleanup, removing old kernels, apt-get clean/autoclean/autoremove to remove unused packages, etc. There is now 2% free space, and the wiki starts.

I wanted to make this a more permenant fix, so I tried to double the HDD space following these instructions. In short, I cloned the drive as a .vdi, enlarged it to 2X the old size, and then used the live CD of gparted to expand the existing drive into the free space. However, when I rebooted the wiki (using the cloned/expanded drive) it still shows the old size. I reloaded the gparted live CD in the VM and double checked; it confirmed that the drive was resized properly.

Anyone know why the new drive size doesn't work & how to fix it?

Thanks

Bryan

 

Forum: 
Support
Tags: 
mediawiki
Jeremy Davis's picture

Jeremy Davis - Tue, 2015/08/18 - 14:48
But you'll need to run it on your actual root mount point. In other words you'll need to find out what root is for your system (meaning the bottom level, aka / - not the /root directory!)

I suggest you try this: 

mount | grep "on / "
It should give a result like this: 
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
or maybe like this: 
/dev/disk/by-uuid/387db0e0-2a50-41e6-85c4-07c4b95d28b6 on / type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)
So then run resize2fs. E.g. 
resize2fs /dev/sda1
or 
resize2fs /dev/disk/by-uuid/387db0e0-2a50-41e6-85c4-07c4b95d28b6
Also regarding cleaning up on your appliance; my strong suspicion is that you have collected a lot of cruft that could be safely removed. There was a bug a few years back that meant that Webmin would dump heaps of cache files and log files into /etc (where settings are normally kept) and because TurnKey also has etckeeper (A neat little app that store everything from /etc in version control) it was not uncommon for /etc to grow out of control... My guess is you have that issue...

I recommend that you install ncdu and use it to have a dig around and see what is taking up all the space...

Install with 

apt-get update && apt-get install ncdu
then launch it with 
ncdu /
Bryan's picture

Bryan - Tue, 2015/08/18 - 15:43

Thanks for the ideas.

 

The resizing didn't work. 

mount | grep "on / "                   

gave me: /dev/mapper/turnkey-root on / type ext4 (rw,errors=remount-ro)

Which I then tried to resize with

resize2fs /dev/mapper/turnkey-root

which gave me:
resize2fs 1.41.11 (14-Mar-2010)                                                                                                                          
The filesystem is already 4456448 blocks long.  Nothing to do!

But when I check the available space the total drive space and the avaiable space remains the same. Anyone know what's going on with that?

 

NCDU showed me the way. I had 14.3 GB in /etc/.git, which is most of the drive. I dug around the forums and  found this command that fixed things up nicely (one of your posts, Jeremy, as I recall):

cd /etc/

git gc

.git went from 14.3GB to 102 MB!

Thanks!

Bryan


Jeremy Davis's picture

Jeremy Davis - Wed, 2015/08/19 - 02:36
I reckon that your system uses LVM (the /dev/mapper/ suggests it). So I'm guessing that when you resized the disk and resized the partition; the logical volume itself hasn't been resized. So I reckon that's what you'll need to do. Liraz actually wrote a great blog post on that. IMO it's worth reading the whole thing; but I have extracted the important bits below. The whole thing is here if you're keen.

So if you were adding 10GB to your current LVM then you would do this: 

lvextend -L+10G /dev/turnkey/root
then do the resize2fs again... 
resize2fs /dev/turnkey/root
PS glad ncdu helped you hunt down your wasted space! :)
Bryan's picture

Bryan - Wed, 2015/08/19 - 15:17

That did the job - probably don't need the extra space now that its all celaned up, but it isn't going to hurt either.

Thanks for walking me through this.

Bryan


Jeremy Davis's picture

Jeremy Davis - Thu, 2015/08/20 - 02:16
Glad we got it all worked out! :)

Add new comment