/lib/modules occupied disk space

And it looks like you have a number of kernels installed. The best (and safest) way to go would be to remove the old kernels.

The first thing to do would be to do a full backup of your system (just in case things go pear shaped).

Then check which kernel you are using

uname -r

I'd hazard a guess that it's the most recent one: 2.6.32-43-generic. Don't uninstall that one! And if you have installed that one recently, I wouldn't uninstall the previous one either (just in case).

So it's a case of uninstalling them one-by-one. You can get the kernel version numbers from what you've listed above. Eg for kernel 2.6.32-26:

apt-get --purge remove linux-image-2.6.32-26-generic linux-headers-2.6.32-26-generic

It is possible the headers aren't installed, if so you can omit that part (ie linux-headers-2.6.32-26-generic). Then just uninstalling them one by one, until you are left with just the one you are currently running (or perhaps 2 if you want to play it safe).

Reboot to test you don't get any nasty surprises later. But only after you have backed up all your important data (if you uninstall the wrong kernel your system may not boot!!!)

Thanks. It works.  After uninstall unused package, the /lib/modules left this:

root@ldap /boot# ls /lib/modules/ -al
total 20
drwxr-xr-x  3 root root  4096 Sep 22 09:28 .
drwxr-xr-x 18 root root 12288 Sep 21 06:56 ..
drwxr-xr-x  4 root root  4096 Sep 18 07:00 2.6.32-43-generic

But the /boot still has these files:

root@ldap /boot# ls /boot -al
total 80868
drwxr-xr-x  3 root root     4096 Sep 22 09:28 .
drwxr-xr-x 22 root root     4096 Sep 22 09:28 ..
-rw-r--r--  1 root root  1690205 Nov 24  2010 System.map-2.6.32-26-generic
-rw-r--r--  1 root root  1690625 Jul  8  2011 System.map-2.6.32-33-generic
-rw-r--r--  1 root root  1691092 Sep 14  2011 System.map-2.6.32-34-generic
-rw-r--r--  1 root root  1691417 Oct 11  2011 System.map-2.6.32-35-generic
-rw-r--r--  1 root root  1694274 Sep  5 17:43 System.map-2.6.32-43-generic
-rw-r--r--  1 root root   651949 Nov 24  2010 abi-2.6.32-26-generic
-rw-r--r--  1 root root   652139 Jul  8  2011 abi-2.6.32-33-generic
-rw-r--r--  1 root root   652395 Sep 14  2011 abi-2.6.32-34-generic
-rw-r--r--  1 root root   652505 Oct 11  2011 abi-2.6.32-35-generic
-rw-r--r--  1 root root   652956 Sep  5 17:43 abi-2.6.32-43-generic
-rw-r--r--  1 root root   116048 Nov 24  2010 config-2.6.32-26-generic
-rw-r--r--  1 root root   116025 Jul  8  2011 config-2.6.32-33-generic
-rw-r--r--  1 root root   116025 Sep 14  2011 config-2.6.32-34-generic
-rw-r--r--  1 root root   116025 Oct 11  2011 config-2.6.32-35-generic
-rw-r--r--  1 root root   116014 Sep  5 17:43 config-2.6.32-43-generic
drwxr-xr-x  3 root root     4096 Sep 22 09:28 grub
-rw-r--r--  1 root root 10009030 Dec 19  2010 initrd.img-2.6.32-26-generic
-rw-r--r--  1 root root 10046359 Jul 22  2011 initrd.img-2.6.32-33-generic
-rw-r--r--  1 root root 10048286 Sep 30  2011 initrd.img-2.6.32-34-generic
-rw-r--r--  1 root root 10048033 Nov  9  2011 initrd.img-2.6.32-35-generic
-rw-r--r--  1 root root 10049358 Sep 18 07:01 initrd.img-2.6.32-43-generic
-rw-r--r--  1 root root     1196 Nov 24  2010 vmcoreinfo-2.6.32-26-generic
-rw-r--r--  1 root root     1196 Jul  8  2011 vmcoreinfo-2.6.32-33-generic
-rw-r--r--  1 root root     1196 Sep 14  2011 vmcoreinfo-2.6.32-34-generic
-rw-r--r--  1 root root     1196 Oct 11  2011 vmcoreinfo-2.6.32-35-generic
-rw-r--r--  1 root root     1196 Sep  5 17:45 vmcoreinfo-2.6.32-43-generic
-rw-r--r--  1 root root  4037888 Nov 24  2010 vmlinuz-2.6.32-26-generic
-rw-r--r--  1 root root  4036896 Jul  8  2011 vmlinuz-2.6.32-33-generic
-rw-r--r--  1 root root  4040128 Sep 14  2011 vmlinuz-2.6.32-34-generic
-rw-r--r--  1 root root  4039584 Oct 11  2011 vmlinuz-2.6.32-35-generic
-rw-r--r--  1 root root  4050048 Sep  5 17:43 vmlinuz-2.6.32-43-generic

Are those old kernel still use?  How to remove them?

Also, how to stop the automatic updating service?

First thing is, if you want a trouble free time going forward, you'll want a (much) bigger root volume IMO. A 2GB root volume is TINY!

2GB on the label, equates to a about 1.8GB of real space (manufacturers calculate 1GB as 1000 x 1000 x 1000 bytes; whereas a computer calculates 1GB as 1024 x 1024 x 1024 bytes). Then by default the installer automatically allocates minimum 512MB as swap. So before you've even started you only have ~1.3GB free space. If you're using LVM, then default is to reserve another 10% for snapshots. Most TurnKey appliance installs take between 700MB and 1.5GB of space once installed, leaving you ~500MB free at best!

Out of interest, I just created a VM with a root volume of 2GB (i.e. really 1.82GB) and installed the v14.2 Fileserver appliance. After install I had less than 300MB free. I then installed the security updates and they failed because it ran out of free space!

Unless you do regular and proactive preventative maintenance (or disable auto-secupdates), I'm not particularly surprised that you've been having these issues. For example, an installed kernel probably takes up around 160MB, with the (compressed) package taking up another ~60MB. That's over 220MB (~10% of your raw space) per kernel update. Given that Debian probably release at least 4 or 5 kernel updates a year (there's been 3 already this year), without intervention that's going to likely be well over 1GB in space used up every year, just on kernel updates!

As a general rule, I use 4GB as my minimum root volume, and that's usually just for a testing VM that will only live a few hours, to a few days. 8-10GB would be my minimum if I expected to use it for anything longer than that. If I wanted to "set and forget" it, the default VM volume size I use is 32GB.

Considering that you can get refurbished hard drives (i.e. second hand, but cleaned and tested) of 80GB+ for as little as US$10, that's probably the way I'd be going. Then it will sit there serving indefinitely! (Or at least until the drive fails - which as with all digital storage is a matter of when, not if - but likely years if the harddrive has plenty of life left in it).

Having said that you could try to squeeze a bit more out of your set up, but even then it will likely continue to require regular maintenance to be reliable. I would recommend culling any software that you don't need (Webmin and Webshell for starters, TKLBAM if you aren't using it, plus probably a number of python modules). Basically try uninstalling stuff with apt and double check that it won't remove anything fundamentally important.

If the machine has a decent amount of RAM, then you could reduce the size of the swap (and then extend the root volume into the free space). If you are using LVM on the SD card, then you extend into the 10% that is likely reserved, and/or allocate some of the space from the USB stick (or from an additional USB stick, etc) and add it to the root volume. Be aware not to remove the wrong stick while it's running though!

Beyond that, you could install ncdu (~100kb installed) to check what is taking up your space. Install it with apt, then launch with the command 'ndcu /':

apt-get update && apt-get install ncdu
ndcu /

It will take a while to scan initially, but then you can easily drill down into directories to see what is taking up the space.

As a general rule, don't ever manually remove anything in /usr (with the exception of /usr/local) as those files are managed by the package management system (i.e. apt/dpkg). If you find a ton of space taken up in /usr that you think you could remove, then work out which package is using it and uninstall it with apt (assuming that you don't want/need the software).

Updated kernels and kernel modules are likely taking up space (although you've likely cleaned them up already). Because of the auto security updates, they will be installed automatically as new kernels with security updates are released. It's also worth keeping in mind that all the auto downloaded security packages will remain in the cache (/var/cache/apt) unless you clear them. You could create a cron job to clean them up automatically. To clean up all security update downloads, run:

apt-get clean

One way to reduce the likelihood of it filling up further would be to disable the auto-security updates. I personally wouldn't recommend that, but if your server is only available within a LAN (i.e. only users/programs on your LAN can access it) that is probably fairly safe. If your server is internet accessible, then I would definitely NOT do that!

I'm not sure what version of TurnKey you're running, but for the most recent release (v14.2) we added a program called monit which can be configured to monitor things such as free disk space. We implemented a very basic config but it should be pre-configured to email when it hits any of the pre-configured alerts (I forget what they were OTTOMH). Although it's possible that unless you've configured an SMTP relay for emails, that it's failing to send the emails. v14.2 also includes an update to confconsole that allows you to easily configure an SMTP relay. It also has a webUI, but as our base config was so simple, we figured that didn't provide a lot of value so disabled it by default. I don't recall the details, but it should be fairly easy to re-enable that if you wish (if it's anything like most Linux utilities, it'll have a config file in /etc, probably a '.conf' file or similar in /etc/monit.

Depending on how old your server is, there are a couple of other likely candidates for chewing up space.

One is the etckeeper's git cache. etckeeper is an awesome app that version controls the whole /etc directory. /etc is where most of the configuration files are stored, so everytime something in there changes, the previous config is saved. Whilst the changes are generally small changes in text files and are saved as a diff (rather than the whole file saved) and compressed, it can still grow significantly over time, especially if there are any binary files saved there (normal diff tools don't work with then so old files need to be saved, plus binary files often don't compress all that well).

Another place that will grow over time are the system logs. Generally they are stored in /var/log. As a general rule it's usually quite safe to remove old logs files, but as they are compressed, unless your system is really old, they won't take up too much space (although my guess is that every MB will count for you). To minimise the space taken by logs in the future, you could tune the relevant logrotate script to be more agressive.

Apologies on the long winded rant, but short story is, get a bigger root volume and then it should "just work" and require less maintenance... If not, some of the measures above may help, but please be prepared for an ongoing maintenance load.

Hi all,
Well, I had the same problem, and this post helped me to solve it temporarily, but I would like a more permanent solution.

I am running a music server with the system on a 2gb SD card, and all the data on a USB disk. Every few months the system disk becomes full and the system crashes before I can find out what happened. I don't carry out any routine maintenance on the server, because I thought it was supposed to be capable of just sitting there serving indefinitely. This time I was able to log on with Putty and saw that I was using 99% of the system disk (Webmin had failed). I can't remember how much space the original installation took, but I know there was plenty to spare. Apart from TKL Fileserver, I have only installed 2 applications, Minimserver and Bubble UPnP server, which I know are pretty small.

I got myself out of a hole and got the aplications running again by running apt-get autoremove, as suggested above, which reduced the disk usage to 90%. So now my questions are:-

1) How to reduce the usage further, since I am pretty sure it was not at 90% when I originally set it up?

2) How to stop the system disk from filling up in the first place?

3) How to monitor disk utilisation so I can spot whether this is happening? The webmin sidebar gives the amount of data on the USB stick, not on the system disk.

Great system when it is working, but it is a PITA when it goes down like this.
Marmite

I find crontab.guru a handy tool for figuring out timings if you wish to use the crontab to trigger a script. However, for your purposes, it may be easier to just create a shell script and drop it into the relevant cron directory (e.g. /etc/cron.daily or /etc/cron.weekly, etc). We have a page on our GitHub wiki dedicated to configuring and testing cron jobs which you may find useful. It provides a bit more info about cron and some useful tips and gotchas.

As per always, if you have any issues, or have some useful advice to share with others, please post back.

Good luck with it all! :)

Thanks again, Jeremy. Quality education, as ever. By running "apt-get autoremove" and "apt-get clean", I have got the disk utilisation down to 75%, so the panic is over for now. You have supplied plenty of ideas for how to tackle the problem more seriously in the longer term. First task will be to find out how to create a cron job which runs the 2 apt-get commands periodically.

Cheers,
Marmite