Media Server for Stability

I know nothing of OMV, so I can't comment too much on that, but I'll do my best to answer your questions. It's also worth noting, that most TurnKey users run it in VMs, so whilst install to hardware should work fine, it's not something that I've ever done. FWIW I run vanilla Debian (currently Stretch) on my work laptop, Debian Sid on my desktop and Proxmox (a Debian based hypervisor) on my server. All my TurnKey servers (including my fileserver/media server) run as LXC containers (mostly) or KVM VMs. I don't think I've personally ever done a bare metal install of TurnKey?!

Also, I'm a commandline guy. I find it much quicker and easier and generally gives much better feedback re what is going on. So I can't really speak with any authority on how to do stuff via Webmin. But hopefully I can point you in the right direction.

Unless you're using USB3 I would imagine that running the OS from USB would be a bottleneck. Although I note that you're trying to limit writes to it, which would certainly help. Personally how I'd do it would be to install the OS to the first disk (on LVM) then shrink the root LV to the desired size, then add the rest of the first drive and all the other disks to a new single LV. But I'm not so familiar with LVM that I could give you the instructions OTTOMH. FWIW when I do stuff like that I usually consult the man page and/or just google.

As you note, TKL is Debian under the hood, so anything Debian should "just work", although as we slim it down as much as possible, some default packages you're used to/expecting may need to be installed.

how to add my existing drives from the webmin, or do I need to mount them via command prompt?

AFAIK you should be able to do that via Webmin, although TBH I'm not sure how that goes. The Webmin docs on Disk and Network Filesystems and Partitions on Local Disks might get you headed in the right direction?

how to merge the FS into one big blob. OMV used mergerfs in the GUI, I can see LVM here is that the way to go?

As I noted above I'd use LVM for this. Webmin has an LVM module, but again, I've never used it, so not sure how intuitive it is.

Although you'll need to do that before you mount and partition the drives (i.e. your previous question). FWIW if you use LVM, you don't even need to partition the drives (assuming that you want to add the whole drive as a "Physical Volume" (the lowest layer of an LVM setup).

implementing SnapRaid for redundancy, OMV did this via the GUI too...

I've never heard of it. I just had a read and it sounds interesting. Although on a drive that is being written to lots, I'm not convinced on how useful it would be in reality (unless you constantly run lots of snapshots, which would add a lot of IO overhead). It'd be fantastic for volumes where very little data changes though (e.g. media storage)! I'm also not sure how it interacts with LVM? If you intend to use it (and are using LVM) then please be sure to research that.

Also, whilst it does provide some degree of redundancy, I can't stress enough that it's NOT A BACKUP! If you have important stuff that you don't want to risk losing, don't rely on snapRAID, do a proper backup!

FWIW LVM can mirror volumes which would give you full redundancy (almost like a backup, but not really...). But that's more like RAID1 so you lose half your storage space, so probably not what you are after...

installing Docker, I ran a few things like Plex...so would like to keep doing that.

I personally don't use Docker much myself (as per my note above about using LXC and Docker and LXC rely on similar backend functionality so they don't really play well together). But as per any Debian system following the Docker docs should "just work".

automating some scripts I've written (I used the server primarily for home movies and photos, so had a script to upload and organise the files)

I assume that you mean via cron jobs? If so, you can do that in Webmin too. Just be a little wary of the fact that cron runs in a minimalist environment, so some environment variables may not be set. As a general rule to avoid issues, use absolute paths when calling commands, your scripts and any other files. Otherwise they may not run.

There is also a cool cron like system called incron. Instead of being triggered by time, it's triggered by filesystem events (e.g. when a file is created). I'm not super experienced with it, but it's possibly worth a look? Not sure if Webmin can handle that though? So you may need to use CLI.

maybe getting my Hauppauge dual tuner working again...one day.

I have no idea there, but assuming that there are suitable drivers that work with Debian, should be doable. Again I can't speak with any authority though.

Re wanting to limit writes to the USB, there are a few ways to do that, but none of them are really ideal in your situation IMO. They would all rely on either mount binding or using an overlay filesystem. And mount an alternate location over the top (e.g. from your HDD or RAM) of the filesystem locations which are often written to (e.g. /var/log, /var/cache, probably some subdirs of /var/lib, etc). You'd need to probably want to have a bit of a google on which dirs should be covered by this (I know that method is often used by Raspbian guys running RaspberryPis with OS installed to an SD card - which have similar issues as you're experiencing with USB sticks).

Using your HDD somewhat defeats the point of installing your OS to the USB IMO (although not completely). Using RAM is possibly better, but that has 2 downsides; you'd have to sacrifice some of your RAM for the purpose and you risk losing your logs if your system crashes (which would likely make troubleshooting really hard). You could occasionally write all these to the USB on a schedule, but there's still potential for issues if something goes wrong between writes (and knowing Murphy's law...).

I'm feeling overwhelmed having to start again. I just want to get my data available for the kids, as they like going through their old memories...that's my main focus at this stage.

I just had a quick google of the OMV forums and found your thread there re problems with your USB install. I can't help but think that your issues are related to your USB and/or the USB setup. I know that you stated that booting with Mint or TurnKey works, but have you tried a full install? (Sorry if you said and I missed it; I only quickly scanned through your posts). Without knowing more about how you set it up and/or what (if any) customisations OMV make to the install routine, I can't really provide any explanation, but my guess is that you may well hit similar issues once you actually install (assuming you haven't yet).

Bottom line is that I'm not convinced that TurnKey is going to be a really great fit for you in your current circumstance and your current knowledge. It's certainly capable of what you want, but you're probably going to need to dig in deeper than you'd probably really like right now...

Despite your recent run of pain with OMV, if you're relatively familiar with it already, happy with it until recently and just want to get up and running ASAP, I think it's possibly your best bet.

Personally, if I were you, I'd be inclined to go buy a small (~100GB) cheap SSD for the OS (should be able to get one ~$100 or less). Not only will that give you a much better experience re the functioning of the OS, it'll also be much more reliable than a USB (unlike USBs, they're designed to be written to lots). Then unplug your existing drives (so you don't risk overwriting anything) then do a clean install of OMV to the SSD. Then power off, plug in your old drives and reboot. If you have any issues booting, make sure that you're booting from the SSD (might require adjustments in BIOS). Then mount/re-add your data drives and configure the other bits. I'm sure the peeps over at OMV will be able to assist with that stuff if you struggle.

If you want to push ahead with TurnKey, I'm happy to assist as best I can, but you'll need to hone your google-fu and the CLI learning curve might be a bit steep. Plus I suspect that you'll likely end up in a similar situation eventually (assuming you keep running from a USB).

Sorry I don't have something better for you...

I just deleted your other (somewhat duplicate) post... It doesn't really matter which subforum it's in as it could apply to either and I'm the main one answering posts here anyway... :)

Yes, I was using USB3, but the current drives were cheap 16GB USB2 drives, for $2 each...just to make sure it wasn't due to faulty drives. It was also easier to mess around with these ones to get it working again. OMV does log to memory with a flash plugin that reduces writes to disk, so in theory they should last a few years, and now that I've backed up the image, I shouldn't have to start all over again next time!

That's the theory at least, I really don't want to have to do this all over again in another 6-12 months... :(

Assuming that this "admin" account you are referring to is a Linux user account, if you log into a shell session as root (either locally, or via SSH) then you can reset the password like this:

passwd admin

That tutorial that you linked to looks pretty good, but I only scanned it, so can't really speak to that in any depth. It's probably a great way to go going, plus you'd likely to get to know Linux better.

Manually running "update-grub" will generally only be useful if you can't boot your system (how you run it if/when you can't boot is another story that I won't go into right now...). If you've been able to boot and haven't been playing with system internals, then it's unlikely to give you any benefit (although OTOH it's unlikely to do any harm either). FWIW grub is the default Debian/Ubuntu boot loader (and is the "norm" for most Linux distros). It's what your UEFI/BIOS hands off to when booting, and it then chainloads the kernel. It needs to know where to find the kernel, which is what 'update-grub' generally does (scans all the drives for kernel). Anyway I digress...

Re your network, unfortunately, I can't give much guidance there (unless you're using TurnKey). TurnKey uses the old legacy way of configuring networking, that has always been solid and reliable and tends to "just work" regardless of whether on bare metal or in a VM, or container, etc. However, I'm well aware that there have been significant changes in the networking config space in more recent times, but I've had limited experience with it. FWIW, there is a "networking" systemd service in more recent Debian releases, but it's not always a good indication of the state of things...

Re the error you link to, it looks like you've since sorted it (perhaps?) but IMO, when troubleshooting issues, it's often best to remove as much ambiguity as possible to try to pin point the issue. So in that case, if there is any chance that a service might be interfering (e.g. Monit in this case) I'd just stop it ('systemctl stop monit'). Beynd that I can't really provide any insight sorry.

Also re SSD, I just had a quick google and found SSDs going as cheap as $40! Personally, I'd get one of those in a flash. The OS will perform so much better (unless you're using USB3 as I mentioned) and the longevity will be vastly better than a USB - so long term price will likely save you plenty (not to mention improved piece of mind). You do mention power usage as a concern too. Intuitively, I'd imagine that an SSD would use less power than a HDD, but again a quick google notes that not as much as I thought... I suspect in the real world, overall they'd use less power, but data suggests not as much as I expected.

Thanks Jeremy. Yes, you're probably right with what you've said. I like working with the USB drives, as it's one less drive to power up (for a machine that's on all the time, and only needs minimal storage size) and it's easy to backup (although I had problems imaging the OMV USB in the past).

That said, maybe I have to look at just going with just vanilla Debian, and doing it all from scratch...I stumbled across this option too: https://blog.linuxserver.io/2017/06/24/the-perfect-media-server-2017/

I actually imagined my OMV4 build, and then just put it back onto the USB and ran it with all drives mounted...and guess what, it booted?!?!?! I'm really not sure what's going on. The only thing I can think, is that I was able to figure out that although I couldn't login to OMV with the admin password, I could log into the box directly with root, and then run update-grub (people kept telling me to do this, but didn't even think to login with root, admin kept dropping back to login prompt). So now I feel more confident, but somehting strange is going on, and it's not configuring the network interface properly (Im' actually wondering, if it never installed properly either, maybe I should choose the other NIC when installing, but I'm too scared now to change anything).

Out of interest, any ideas what could be causing this one: https://forum.openmediavault.org/index.php/Thread/23276-Cannot-create-socket-to-localhost-2812-Connection-refused/

I appreciate your help and advice, I think I will keep trying with OMV now, but I have just taken such a huge hit in confidence with all the troubles that I've been having trying to install it.

I'm glad to hear that it was at least of some value to you. And even though TurnKey probably isn't the system for you, thanks for having a look anyway. Good luck with it all.

Thanks again for sparing thte time to provide some much needed coaching and advice, it kept me going :)