Marmite Sandwich's picture

So I got my music server running 9 months ago, as a newbie. During that time, I didn't need to do anything with TKL (as planned) so I am still a newbie! A problem developed with the external USB drive, which I have now replaced. In the process of rather clumsy unmounting, remounting, testing and rebooting via the web interface, I have ended up with a device which the system is looking for, but which is not there. When booting up, the system waits for 1.5 minutes for a response from this non-existent device and then goes into emergency mode and the web interface is not available. How do I prevent the system going into emergency mode under these circumstances, and how can I get out of it?

I have entered the root password which allowed me to enter console commands: "service webmin status" confirmed that the service is inactive (dead). "Service webmin stop" and "service webmin start" returned "job for webmin.service canceled", after which the keyboard locked up. So much for emergency mode. If I can just get webmin going, I can reconfigure the mounted devices appropriately and subsequent reboots won't hang.

All suggestions gratefully received.
Marmite

Forum: 
Jeremy Davis's picture

Generally "emergency mode" is intended for use via commandline only. It generally loads what is known as "single user mode". That means that the system allows only root login and service availability will (by design) be limited. It's like that as the less things running, the better chance you have of being able to log in and fix the underlying issue.

My guess is that you have added the previous USB device via the fstab. If I'm right (which I'm pretty sure of), then you should be able to resolve it really easily.

When logged in at the commandline, please try opening the fstab file and commenting out the line that mounts your external USB. Then reboot. The commands to do that would look something like this:

# open the fstab for editing
nano /etc/fstab

# locate the line that mounts your USB device then comment it out.
# i.e. add a '#' character at the start of the line; like these lines
# save the file and exit nano (Ctrl-X IIRC)

# check that everything from the updated fstab mounts properly
mount -a

# if things still aren't right, re-edit the fstab as per above
# once everything appears ok, reboot
reboot
Hope that helps
Marmite Sandwich's picture

Hi Jeremy and thanks again for your help. I am using TKL because it has a web interface! I don't mind learning the odd command and "lifting the hood" occasionally (and documenting it in my notes for 9 months time when I need to use it again), but I am trying to avoid learning another command set just for my music server.

Anyway, I managed to get around this problem by attaching my USB device after booting had started, but before it reached the point where the system was looking for the device referenced in the saved mount. My diagnosis of the problem is this. The storage device reference which is allocated if you attach it when the system is running is different from the reference it gets from the BIOS if you attach it before you set the boot device priority at startup. So I should have unmounted the old storage device, removed the save on the mount, shut down, attached the new device, set the boot priority in the BIOS, booted and THEN mounted the new device, with a save.

So then I checked out your fstab solution in case it happens again for some other reason. I located and opened up fstab using WinSCP and it looks like this:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda1 during installation
UUID=d1e31bd1-5c07-497c-a97a-f190670c41aa /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=532a5129-1d54-42d4-b60e-5528bc892c4c none            swap    sw              0       0
/dev/sda1    /externalusb    ntfs    defaults    0    0

TKL and a swap partition are on a 4GB SD card (sda5 above?), and the data is held on the new USB device (sda1?). I don't see a "mount -a" type of line that can be commented out. Can you suggest how I could go about modifying this file from emergency mode, assuming I can run nano and want to unmount all storage devices apart from the one with the system on it? It looks like I need to comment out the "UUID=" line with the reference to ext4 in it, but I am only guessing.

Much appreciate your help.
Marmite

Jeremy Davis's picture

You're most welcome for the help. That's what I'm here for! :)

Personally, I find the commandline much easier to work with than a GUI. Although all my computers run Linux and I play with servers a fair bit (as you'd imagine) so I understand that a GUI often feels easier when you are unfamiliar with commandline. I'm not sure why, but many users used to using a GUI are a little scared off by commandline. As a general GUI user (back when I used Windows all the time), I too was a little scared off my commandline. But I found that once I got over the anxiety, it's actually awesome and heaps easier to use than a GUI (I still use the GUI lots within my Linux desktop, but there lots of things that I only do via commandline). But I won't try to convert you just yet, I'll just try to help you out! :)

So back to the issue at hand.

The 'mount -a" command is to test your fstab file after you have edited it (more on that further down). As I hinted above, any line that starts with '#' is considered a comment (and is ignored). Also if you are interested in understanding more, I suggest you have a look at the Debian fstab wiki page. Anyway, let me remove most of the comments, abbreviate/abridge the UUIDs and realign the text, so the info may make some more sense to you (teach a man to fish and all that! :)

# <file system>     <mount point>    <type>    <options>          <dump>  <pass>
UUID=d1e31bd1-[...]   /               ext4    errors=remount-ro     0       1
UUID=532a5129-[...]   none            swap    sw                    0       0
/dev/sda1             /externalusb    ntfs    defaults              0       0

The best way to work out which device (called a 'file system' in fstab table) is the one to comment out, start at the 'mount point'. You can see there that the first one (UUID=d1...) is mounted to '/' and has a type of 'ext4' (standard/default Linux filesystem) so that's the main volume for your server, i.e. the main/base directory that contains everything! So don't comment that one out! Your system won't boot at all!

The next one (UUID=53...) is not mounted at all (mount point 'none') which seems weird, but if you look at 'type' it's 'swap'. Swap on Linux is essentially the same as the pagefile on Windows, i.e. "virtual memory" (basically it uses part of the harddrive as extra memory if the system runs out). But instead of using a file (like Windows does) Linux uses a partition (it can use a file too, but a partition is better, actually I think Win10 now defaults to a paging partition?). So leave that one be too!

So by a process of elimination, your external USB should be the last line, the one that starts with '/dev/sda1'. And if you look at both the 'mount point' and the 'type', that confirms things. It's mounted to '/externalusb' (sounds likely!) and has a filesystem type of "ntfs" (default Windows file system).

So comment that line out. I.e. put a '#' at the start, so the line now looks like this (I've compressed the spaces so that it displays better, you don't need to do that, just add the '#' at the start):

#/dev/sda1          /externalusb ntfs    defaults          0    0

Because we've only made a minor change that I'm almost certain will be correct, there probably isn't any need to test it. However, it's good practice (especially when you're not 100% confident). So double check that your fstab is right by running:

mount -a
That will try to mount everything that is in your fstab. You'll get an error message if there is something wrong with your fstab.

As a bit of an aside, it's a little weird that your USB drive was appearing as /dev/sda as that is usually the first hard drive. Normally that would be your root drive. However, that's why they use those weird looking long "UUID-..." labels. As it says in the comment:

# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
That's why your system still continued to work. If it used /dev/sda1 as the root ('/') device (instead of UUID=d1e31bd1-5c07-497c-a97a-f190670c41aa) your system would not boot when your USB drive was plugged in!
Marmite Sandwich's picture

Is it possible to configure the system so that you can define what constitutes an emergency? For me, that wouldn't include the inability to mount an expected storage device.

Marmite

Jeremy Davis's picture

The reason why it freaks out when a drive listed in the fstab isn't available is because the expectation is that only permanent drives are listed in the fstab (unless clearly specified - which I'll get to in a sec). So if one of those "permanent drives" is not available at boot, the system assumes that something really bad has happened! Like a failed harddrive or something! And I'm not sure about you, but personally I think that if a system drive isn't available at boot time, then something really bad has happened!

Not that there is anything wrong with permanently mounting an external drive, but as you can see, it has implications!

If you wish to have your system boot, whether the external USB is connected or not, then there are fstab option to support that. The one to add to allow the computer to boot properly if the drive isn't attached is 'hotplug'. Another useful one to include is 'auto' (which means it will mount when you run 'mount -a'). So the fstab line for your external USB would become something like this:

/dev/sda1         /externalusb    ntfs    auto,hotplug,defaults           0       0

Although as noted above, rather than using '/dev/sda1', you should really find out it's UUID and use that instead. That way it will only ever mount the specific external USB to the desired place (and not just any random device that happens to be '/dev/sda1'). FWIW, if you'd rather not use UUIDs, you can also use disk labels (i.e. 'LABEL=USB_DISK_LABEL' - replacing USB_DISK_LABEL with the actual disk's label - set at format time).

Note though, that that won't take care of unmounting when you remove the drive while the server is still running. But at least it will set reasonable disk write caching defaults so the chance of data being unwritten when you remove the disk is reduced.

If your server is generally "always on", then you can just manually mount and unmount as required. I'm pretty sure you can do that via Webmin, but I can't recall. Doing that would look something like this:

# to mount:
mount -a

# to unmount (command is 'umount' i.e. without the 'n')
umount /externalusb

Another, probably more user friendly option, is to configure auto mount for the drive when it's plugged in. That is the default behaviour on a Linux desktop system (i.e. similar to Windows and Mac OSX). Please be careful though as that can cause data corruption if you remove the drive while it's still mounted. You can set options to reduce the chance of that happening, but if the drive is removed while data is being written to it, then it can still cause corruption.

As TurnKey is designed as a somewhat minimalist headless server distro, we don't include that to keep the size down. Although perhaps we should consider including it for the fileserver based appliances in the future? It's probably something that users would find handy.

Unfortunately, I'm a bit out of touch with how to do all that, but a quick google suggests that the package you want to install is 'udisks2'. Unfortunately, I haven't played with and can't be sure how it's used. If you want to have a play with it, I suggest that you install it ('apt-get update && apt-get install udisks2) and go mad! :)

Here are a couple of links that may be handy:

  • https://docs.ubuntu.com/core/en/stacks/disk/udisks2/docs/reference/snap-configuration/automount

    Note that the first is for Ubuntu. Ubuntu is based on Debian (as TurnKey is) so generally instructions that apply to Ubuntu, will also apply to TurnKey (and other Debian derivatives) although please be aware that Ubuntu is not binary compatible with Debian (TurnKey is), so it's best to NEVER add Ubuntu specific repositories (e.g. PPAs). It's also worth noting that Ubuntu is produced from snapshots of Debian unstable, so will almost always have newer version of packages than TurnKey.

  • https://wiki.archlinux.org/index.php/udisks

    Note, this one is for a completely different Linux distro called Arch. It's a bleeding edge distro, so it's docs are always going to be for newer versions of software than what TurnKey has. However, IMO it has hands down the best Linux documentation of any distro! So I often find it incredibly useful, even though you do sometimes hit weird issues (i.e. use options and config that aren't relevant to the software version installed on TurnKey).

    Good luck! And if you do end up working out auto mount and unmout (e.g. via udisks2) please let me know. As I say, it might be cool to add that by default to the fileserver (and media server, etc)?!

  • Marmite Sandwich's picture

    That's a great information resource right there. I will save it and reference it as my system becomes more sophisticated. My strange file system references are probably caused because I have no conventional hard drive. The system is on a SD card and the data is on a USB stick. The hard drive bay is empty.

    Thanks again for your educational support.
    Marmite

    Add new comment