rzwemer's picture

Unable to connect via https://192.168.1.82:12321/ or https://192.168.1.82:12320/, but can using Putty with SSH. Have searched the forums but unable to find anything. Have 2 servers setup one at 192.168.1.81 works fine the other at 192.168.1.82. Able to look around with SSH, but don't know what to look for to correct this.

Have added screen shots for both Webmin, Terminal & SSH for both servers.

Thank you in advance for any help, will continue to search

rzwemer

webshell working

webshell not working

Webmin working

Webmin not working

ssh working

ssh working

Forum: 
Jeremy Davis's picture

First up, these are really old servers. If they are available online (i.e. you are publicly hosting stuff on the open internet) then I would highly recommend that you prioritise migrating your data to a newer release or upgrading your OS.

Back to your issue at hand, I'm a bit rusty WRT v14.2 but I suggest checking whether the services are running. Try this:

systemctl status webmin
systemctl status shellinabox

In more recent TurnKey releases, both Webmin and Webshell are behind stunnel. I don't think that was the case in v14.2, but I think Webshell might have been?! Perhaps have a look:

systemctl list-units | grep stunnel

You can also see what is listening on which ports like this:

netstat -tlnp

Look for 'shellinaboxd' (Webshell), 'stunnel4' (or perhaps just 'stunnel') & 'perl' (Webmin simply reports itself as 'perl'). Feel free to post the output if you are unsure and want my input.

If they aren't running, then I suggest trying to start them and see what happens:

systemctl restart webmin
systemctl restart shellinabox

If you had an stunnel service, then try starting/restarting that too.

If that works, then I'd personally reboot the server to make sure Webmin and Webshell come back up again after a reboot. If you continue to have issues, please post back output of the status commands.

rzwemer's picture

Jeremy Davis,

Thanks for the prompt response and advise.

You are correct about being old servers, guess that "If it ain't broke don't fix it" does not apply here. Have followed your advice, results are: status webmin & shellinabox both active (running). Did restart the server, also ran: apt update, only to discover: no space left on server.

Noticed for the server which is accessible it has an encrypted connection, the other does not.

Have several more screen shots but unable to add them.

rzwemer

Jeremy Davis's picture

(FWIW I wrote this earlier but just realised that I didn't post it)

Your screenshot didn't render. I had a quick look to see if I could fix it, but it appears it's in base64 format, rather than being a link to a file. I tried converting the base64 back to a file, but it didn't work. As I noted in my other post, copy/pasting the CLI input/output is generally better (obviously screenshots are useful for showing things within the GUI).

If you do want/need to post more screenshots, either upload them to a third party service, or edit the initial post and attach them there (only the first post of a thread can have attachments). If you do add them to your OP, then either display them (or even just the link to them) or at least let me know. That's because they aren't visible by default (I know it's dumb).

Regardless, from the subject line of this post, I assume that both webmin & shellinabox are running. If that's the case and it still isn't working, please double check netstat ('netstat -tlnp') and paste the output for me.

Jeremy Davis's picture

You haven't confirmed, but I suspect that once you have your free space issue resolved, you will likely be "good to go"!

As for "If it ain't broke don't fix it", so long as they aren't publicly available, then that's certainly not completely unreasonable.

But it is worth being aware that you won't find many updates via apt. Debian Jessie (the basis of v14.x) is still somewhat supported via ELTS (extended long term support) for a couple more months (support is slated to completely end this June). But ELTS only has supports the specific packages requested/required by the sponsors. That means that whilst everybody is free to use it, unless you are a paying subscriber, there is no guarantee that all the packages you are using will get security updates.

If you intend to keep these servers running, then I would encourage you to consider some maintenance. How viable that is will depend on the software you are running and what dependencies it has. If you care to share a bit more about what you have running and it's purpose, I'm more than happy to give you my 2c.

Re screenshots, TBH, copy/pasting from your terminal is probably better. Obviously that doesn't work for web pages, but terminal stuff should work fine. It looks like you are using PuTTY, simply selecting text within a PuTTY session should copy it (right click to paste into a PuTTY session).

If you need a hand resizing/extending your server root volume, then assuming that your servers are installed using LVM (default options when installing from ISO, included in VMDK & OVA builds), please have a look at this TurnKey blog post. It's really old, but still remains relevant. If you have any issues please ask.

rzwemer's picture

Jeremy,

Again thanks for the speedy response, have to say I'm not used to this kind of of detailed help.

These servers are not publicly available, just internal for my own use, mainly storing still pictures to be made into time lapse videos. Do intend to keep these servers running, these are TurnKey File Server installs with no mods, would appreciate your 2c.

Following are the requested screen shots:

login as: root

root@192.168.1.82's password:
Welcome to Fileserver02, TurnKey GNU/Linux 14.2 / Debian 8.8 Jessie

  System information (as of Fri Apr 15 18:34:03 2022)

    System load:  1.42               Memory usage:  5%
    Processes:    119                Swap usage:    0%
    Usage of /:   100.0% of 5.37TB   IP address for eth0:  192.168.1.82

  TKLBAM (Backup and Migration):  NOT INITIALIZED

    To initialize TKLBAM, run the "tklbam-init" command to link this
    system to your TurnKey Hub account. For details see the man page or
    go to:

        https://www.turnkeylinux.org/tklbam

Last login: Fri Apr 15 18:13:03 2022 from 192.168.1.90

root@FILESERVER02 ~# systemctl status webmin
* webmin.service - LSB: Webmin
   Loaded: loaded (/etc/init.d/webmin)
   Active: active (running) since Fri 2022-04-15 18:33:06 UTC; 23h ago
  Process: 585 ExecStart=/etc/init.d/webmin start (code=exited, status=0/SUCCESS                                                                             )
   CGroup: /system.slice/webmin.service
           `-878 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/mini...

Warning: Journal has been rotated since unit was started. Log output is incomple                                                                             te or unavailable.

root@FILESERVER02 ~# systemctl status shellinabox
* shellinabox.service - LSB: Shell In A Box Daemon
   Loaded: loaded (/etc/init.d/shellinabox)
   Active: active (running) since Fri 2022-04-15 18:32:56 UTC; 23h ago
  Process: 588 ExecStart=/etc/init.d/shellinabox start (code=exited, status=0/SU                                                                             CCESS)
   CGroup: /system.slice/shellinabox.service
           |-700 /usr/bin/shellinaboxd -q --background=/var/run/shellinaboxd....
           `-701 /usr/bin/shellinaboxd -q --background=/var/run/shellinaboxd....

Warning: Journal has been rotated since unit was started. Log output is incomple                                                                             te or unavailable.
                                                                   -
root@FILESERVER02 ~# netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      709/sshd
tcp        0      0 0.0.0.0:47517           0.0.0.0:*               LISTEN      764/rpc.mountd
tcp        0      0 0.0.0.0:33758           0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:12319         0.0.0.0:*               LISTEN      700/shellinaboxd
tcp        0      0 0.0.0.0:47104           0.0.0.0:*               LISTEN      764/rpc.mountd
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:50722           0.0.0.0:*               LISTEN      565/rpc.statd
tcp        0      0 0.0.0.0:35656           0.0.0.0:*               LISTEN      764/rpc.mountd
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      556/rpcbind
tcp        0      0 127.0.0.1:10000         0.0.0.0:*               LISTEN      878/perl
tcp6       0      0 :::22                   :::*                    LISTEN      709/sshd
tcp6       0      0 :::443                  :::*                    LISTEN      788/apache2
tcp6       0      0 :::2049                 :::*                    LISTEN      -
tcp6       0      0 :::32964                :::*                    LISTEN      764/rpc.mountd
tcp6       0      0 :::51434                :::*                    LISTEN      764/rpc.mountd
tcp6       0      0 :::50127                :::*                    LISTEN      565/rpc.statd
tcp6       0      0 :::111                  :::*                    LISTEN      556/rpcbind
tcp6       0      0 :::80                   :::*                    LISTEN      788/apache2
tcp6       0      0 :::42001                :::*                    LISTEN      764/rpc.mountd
tcp6       0      0 :::59188                :::*                    LISTEN      -
root@FILESERVER02 ~#

root@FILESERVER02 ~# systemctl list-units | grep stunnel
  stunnel4.service                                                                                           loaded active exited    LSB: Start or stop stunnel 4.x (SSL tunnel for network daemons)
root@FILESERVER02 ~#

rzwemer

Jeremy Davis's picture

TBH, encouraging people to get more comfortable with Linux CLI and/or resolve their issues is something of a passion of mine! So whilst you are welcome, it's kinda my thing... :)

Back to your issue, it looks to me like you need to start stunnel (it's actually called stunnel4). So the TL;DR answer is try this:

systemctl start stunnel4

If that doesn't work, please share the output of:

systemctl status stunnel4

And we'll take it from there...


But in the interests of teaching a man to fish and all that...

I get there because I know that stunnel essentially acts as a SSL terminating reverse proxy (an intermediary between the local service and the external internet). I initially didn't recall whether that was the case in v14.x or not (as it turns out, it was introduced in v14.0), but what you've posted has confirmed my suspicion and refreshed my memory (plus I double checked by looking back through TurnKey changelogs).

To be clear, you weren't/aren't necessarily expected to know that. AFAIK it's a config that's unique to TurnKey. I did want to mention it though, to clarify that even beyond my pretty good general Linux knowledge, I have an advantage as I'm intimately familiar with TurnKey! :)

But to demonstrate how what you shared help me confirm my suspicions, let me explain a little more...

The output of the first 2 systemctl commands shows that both shellinabox (aka Webshell) and Webmin services are running (note the line in each: " Active: active (running)" - if it wasn't running IIRC it would be something like 'active (exited)" or perhaps "failed"/"stopped" or similar).

The output of netstat confirmed my suspicions. Note that 'netstat -tlnp' shows all services that are listening to the network and which particular port and interface they're listening on. As you can see from your output, there are no stunnel services (as noted by the heading; the far right column is "PID/Program name"; PID = Process ID) and both shellinabox and webmin (878/perl in your output - note that that PID corresponds with what is displayed within the Webmin systemctl output) are only listening on localhost. So not only are there no stunnel processes listed listening, there is nothing listening on ports 12320 or 12320!

As something of an aside, just in case you weren't aware, "Local Address" 0.0.0.0 means "all interfaces" (i.e. any/all NICs attached) and "127.0.0.1" is localhost only (i.e. no external connections).

As per your previous info, we know that running out of free space is the likely cause of stunnel originally failing. So in theory, with the cause resolved, stunnel should "just work". However, there may be something else wrong. So if it doesn't "just work", then hopefully the info from "systemctl status stunnel4" might give us some insight into why.

rzwemer's picture

Jeremy,

When first installed, TurnKey File Server, several years ago, it performed flawlessly,
which probably caused some laziness on my part again "If it ain't broke don't fix it".

Not able to add more storage, so is it possible to move or delete enough storage via
SSH with Putty to allow stunnel4 to start, then can move/backup current store and then
upgrade to current version of Turnkey File Server?


After starting stunnel4, then checking status, still unable to connect, here are results:

root@FILESERVER02 /dev# systemctl start stunnel4
root@FILESERVER02 /dev# systemctl status stunnel4
* stunnel4.service - LSB: Start or stop stunnel 4.x (SSL tunnel for network daem                                                                             ons)
   Loaded: loaded (/etc/init.d/stunnel4)
   Active: active (exited) since Fri 2022-04-15 18:32:56 UTC; 3 days ago
  Process: 613 ExecStart=/etc/init.d/stunnel4 start (code=exited, status=0/SUCCE                                                                             SS)

Apr 19 14:15:19 FILESERVER02 systemd[1]: Started LSB: Start or stop stunnel ....
Warning: Journal has been rotated since unit was started. Log output is incomple                                                                             te or unavailable.
Hint: Some lines were ellipsized, use -l to show in full.
root@FILESERVER02 /dev# systemctl stop stunnel4
root@FILESERVER02 /dev# systemctl status stunnel4
* stunnel4.service - LSB: Start or stop stunnel 4.x (SSL tunnel for network daem                                                                             ons)
   Loaded: loaded (/etc/init.d/stunnel4)
   Active: inactive (dead) since Tue 2022-04-19 14:40:21 UTC; 6s ago
  Process: 13306 ExecStop=/etc/init.d/stunnel4 stop (code=exited, status=0/SUCCE                                                                             SS)
  Process: 613 ExecStart=/etc/init.d/stunnel4 start (code=exited, status=0/SUCCE                                                                             SS)

Apr 19 14:15:19 FILESERVER02 systemd[1]: Started LSB: Start or stop stunnel ....
Apr 19 14:40:21 FILESERVER02 systemd[1]: Stopping LSB: Start or stop stunnel....
Apr 19 14:40:21 FILESERVER02 stunnel4[13306]: Stopping SSL tunnels: stunnel.
Apr 19 14:40:21 FILESERVER02 systemd[1]: Stopped LSB: Start or stop stunnel ....
Warning: Journal has been rotated since unit was started. Log output is incomple                                                                             te or unavailable.
Hint: Some lines were ellipsized, use -l to show in full.
root@FILESERVER02 /dev# systemctl status stunnel4
* stunnel4.service - LSB: Start or stop stunnel 4.x (SSL tunnel for network daemons)
   Loaded: loaded (/etc/init.d/stunnel4)
   Active: inactive (dead) since Tue 2022-04-19 14:40:21 UTC; 59s ago
  Process: 13306 ExecStop=/etc/init.d/stunnel4 stop (code=exited, status=0/SUCCESS)
  Process: 613 ExecStart=/etc/init.d/stunnel4 start (code=exited, status=0/SUCCESS)

Apr 19 14:15:19 FILESERVER02 systemd[1]: Started LSB: Start or stop stunnel 4.x (SSL tunnel for network daemons).
Apr 19 14:40:21 FILESERVER02 systemd[1]: Stopping LSB: Start or stop stunnel 4.x (SSL tunnel for network daemons)...
Apr 19 14:40:21 FILESERVER02 stunnel4[13306]: Stopping SSL tunnels: stunnel.
Apr 19 14:40:21 FILESERVER02 systemd[1]: Stopped LSB: Start or stop stunnel 4.x (SSL tunnel for network daemons).
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root@FILESERVER02 /dev# systemctl start stunnel4
root@FILESERVER02 /dev# systemctl status stunnel4
* stunnel4.service - LSB: Start or stop stunnel 4.x (SSL tunnel for network daemons)
   Loaded: loaded (/etc/init.d/stunnel4)
   Active: active (exited) since Tue 2022-04-19 14:41:31 UTC; 6s ago
  Process: 13306 ExecStop=/etc/init.d/stunnel4 stop (code=exited, status=0/SUCCESS)
  Process: 13363 ExecStart=/etc/init.d/stunnel4 start (code=exited, status=0/SUCCESS)

Apr 19 14:41:31 FILESERVER02 stunnel[13376]: LOG5[13376]: stunnel 5.06 on x86_64-pc-linux-gnu platform
Apr 19 14:41:31 FILESERVER02 systemd[1]: Started LSB: Start or stop stunnel 4.x (SSL tunnel for network daemons).
Apr 19 14:41:31 FILESERVER02 stunnel4[13363]: Starting SSL tunnels: [Started: /etc/stunnel/stunnel.conf] stunnel.
Apr 19 14:41:31 FILESERVER02 stunnel[13377]: LOG3[13377]: Cannot write pid file /stunnel4.pid
Apr 19 14:41:31 FILESERVER02 stunnel[13377]: LOG3[13377]: write: No space left on device (28)
root@FILESERVER02 /dev#

rzwemer

 

Jeremy Davis's picture

That 100% confirms that lack of free space is the cause! So I guess that's progress of a sort...

TBH, in retrospect, I probably should have made the connection between stunnel not starting and no free disk space much sooner. I think it was the fact that so much other stuff seems to be running fine that threw me a bit. And looking back over this thread, I should have noticed that in the relevant screenshot of your OP, clearly notes "Usage of /: 100%".

Anyway, you say that you can't add anymore space to the server? Why is that? Is it baremetal and you have no spare physical HDD to add? Or is it a VM with no more free space on the host? Please share how it's installed and what platform it's running on. I might have some ideas for you...

To answer your question explicitly, one super easy way to clear a bit of free space is clearing out the apt cache:

apt-get clean

That should hopefully get you a few MB at least, possibly much more.

Another way that you can likely clear some space is removing old kernels. That is a little more convoluted (and you need to be careful - removing the wrong thing will brick your server - not irreparably, but it'll make this process a whole world of pain...).

List the current kernel you are running like this:

uname -r

Whatever you do, don't remove that one! FWIW, here's what my laptop (running Debian) returns (yours will be much older obviously):

5.10.0-13-amd64

Then you can check which kernel packages are installed like this

dpkg --list | grep linux-image | grep -v awk '{print $1" "$2}'

Again here's my output:

rc linux-image-4.19.0-17-amd64
rc linux-image-4.19.0-18-amd64
rc linux-image-5.10.0-11-amd64
ii linux-image-5.10.0-12-amd64
ii linux-image-5.10.0-13-amd64
ii linux-image-amd64

In my example, the last one (linux-image-amd64) is the "meta package". That one needs to stay - always, as does the one that's currently being used. If you compare my uname output above against this list, you can see that I'm running the latest kernel (linux-image-5.10.0-13-amd64) so I won't be removing that one. If your uname output doesn't match the most recent kernel installed, reboot first (and double check that you're running the newest kernel once rebooted).

As you can see, my top 3 have an 'rc' prefix. This means that they've already been removed ('r') but still have config remaining ('c'). 'ii' means it's installed. So in my case all of these can go, except for the last 2 (i.e. NOT linux-image-5.10.0-13-amd64 or linux-image-amd64). To do that, use 'apt-get remove' to remove the package and to clear the old config, use the '--purge' switch. So here's what I would run:

apt-get remove --purge -y linux-image-4.19.0-17-amd64 linux-image-4.19.0-18-amd64 linux-image-5.10.0-11-amd64 linux-image-5.10.0-12-amd64

(In case I haven't already made it clear, you need to use the names from your output)

Towards the end of the output, there may be warnings that directories aren't empty, so not removed. E.g. here's what mine noted:

Purging configuration files for linux-image-5.10.0-11-amd64 (5.10.92-2) ...
rmdir: failed to remove '/lib/modules/5.10.0-11-amd64': Directory not empty
dpkg: warning: while removing linux-image-5.10.0-11-amd64, directory '/lib/modules/5.10.0-11-amd64' not empty so not removed
Purging configuration files for linux-image-4.19.0-18-amd64 (4.19.208-1) ...
rmdir: failed to remove '/lib/modules/4.19.0-18-amd64': Directory not empty
dpkg: warning: while removing linux-image-4.19.0-18-amd64, directory '/lib/modules/4.19.0-18-amd64' not empty so not removed
Purging configuration files for linux-image-4.19.0-17-amd64 (4.19.194-3) ...
rmdir: failed to remove '/lib/modules/4.19.0-17-amd64': Directory not empty
dpkg: warning: while removing linux-image-4.19.0-17-amd64, directory '/lib/modules/4.19.0-17-amd64' not empty so not removed
Purging configuration files for linux-image-5.10.0-12-amd64 (5.10.103-1) ...
rmdir: failed to remove '/lib/modules/5.10.0-12-amd64': Directory not empty

If you don't get that, that's fine. If you do get something similar, you can manually remove those if you want (they're unlikely to be large, so your call). Here's what I did:

rm -r /lib/modules/4.19.0-17-amd64 /lib/modules/4.19.0-18-amd64 /lib/modules/5.10.0-11-amd64 /lib/modules/5.10.0-12-amd64

One last place where you can easily clean up a bit more space is removing old logs from is in the logs directory, /var/log. If you have a poke around in there, you should see a lot of files with similar names, just a different suffix (generally either '1', or a number and '.gz' suffix. Those files can all be deleted (don't delete the ones that are just '.log' files or have no extension.

It's likely that it won't give you a ton of extra space, but should clear at least a few MB and it won't do any harm. Here's a quick way to delete them all:

find /var/log -regextype posix-extended -regex '.*\.[1-2]?[0-9][\.gz]?' -exec rm {} +

I'm happy to explain the regex if you're interested.

If you follow all those steps, that should give you plenty enough free space to get you going again. It may even allow you to be happy to leave it be for now (it's already so old that it'll be a bit of a pain to upgrade/migrate so as noted before, if it's working, you could leave it be.

If you'd like to clear more space, I suggest that you install ncdu (a tiny interactive CLI tool for browsing through the filessytem). Install like this:

apt-get update
apt-get install -y ncdu

Then use ncdu to explore your filesystem, like this:

ncdu /

It will start at the root of your filesystem and will sort directories and files by size and allows you to drill down and see what is using the space. Obviously you'll need to be a bit careful, as if you start randomly deleting stuff you could destroy your server! As a general rule, don't delete anything from /boot /bin /dev /lib /lib64 /proc /run (if it exists) /sbin /sys or /usr (/usr/local is something of a special case; you can often delete stuff there, although if you don't know what it is, don't...). Also be careful in /etc and /var. Files in /home (normal user home dirs) and /root (root's home dir) can be removed, but "dot files" (files and/or directories starting with a '.') are user config related, so again be careful. If you're unsure, ask...

If you're thinking of upgrading anyway, then perhaps another way to go is to create a new VM (using a newer TurnKey) and copy across your files. You could use rsync to transfer the files directly between the 2 servers, or TKLBAM (but don't do an automated restore, you'll need to do a "staged" one for best result).

Anyway, good luck with it all and look forward to hear how you go.

rzwemer's picture

Jeremy,

Things are looking up, took your advice and removed excess log files ending in .n or .n.gz (n as 1 - 9), then initiated: systemctl start stunnel4. Now able to access over my network via samba and copy and delete files. Still unable to access webmin or shellinabox thru a browser. Will spend the next several days moving and removing files. Once either of the current TurnKey File Servers is cleaned, will be looking to upgrade it to the latest version of TurnKey File Server. Had an 8 TB external hard drive available from a previous mishap with another server (Ubuntu server with 4 x 2 TB WD Red NAZ Storage) running RAID 5.

By the way I truly appreciate all the help, I'm 75, retired/semi-retired about 15 year ago, worked in IT, started out with Fortran and COBOL, since learned some new stuff. Now mostly dink with 3D Printers , CNC machines and a lot of Raspberry Pi's, with a sprinkling of arduino, ESP8266 etc.

Bob (rzwemer)

P.S. Again can't thank you enough for the help, will probably be asking for assistance again in several days.

Jeremy Davis's picture

So long as both stunnel and Webmin/Shellinabox are running, then in theory it should "just work"! Feel free to share current status info if you want some input decoding the logs.

Ah Fortran and COBOL eh!? I've never used either, but I recall talk of them from when I was young! :) I played with basic a bit and recall being in awe of "real programmers" that worked their wizardry with code like that! :)

My son has a 3D printer, but I haven't played with it much (I like the idea, but the learning curve to make anything useful is higher than I have spare cycles for for now). I also have some old Rpis and heap of ESP8266s that I had grand plans for (which are yet to actually come to anything...).

Actually, speaking of Raspberry Pi, do you happen to have a pi4? If you do and you have a spare sd card and a little time, it'd be super awesome if you could have a look at the "preliminary" Rpi4 TurnKey builds. They're not "official" but Yannick (the guy that's built them) is a long time TurnKey community member and it would be awesome if we could give him some feedback and push that forward. I've just bought one and hope to have a play so I can test them out too. Hopefully we can take a few of the remaining sharp edges off and make them official some time soon! :)

rzwemer's picture

Jeremy,

You just need to retire then initially you will have plenty of spare cycles, but that only lasts a month or so then you will have fewer spare cycles than when you worked.

The RPi 4 with 8GB can replace most end user computers, even with 4GB like the RPi 400.

As I'm moving & removing my files, I've come to the conclusion, I'm a bit of a "DIGITAL HOARDER", it's going to take some time to sift through my hoard and get rid of the chaff, so checking out Yannick's ports will give me some relief from the tedium.

Will keep you posted of my progress and plan to upgrade to the current version of Turnkey File Server.

Enjoy your weekend.

Bob (rzwemer)

Jeremy Davis's picture

Good luck with your digital spring cleaning! I can certainly relate... :)

rzwemer's picture

Jeremy,

Was able to speed up the data transfer 10 fold, connected a laptop and the Turnkey File Server via GB ethernet to a router, with the external HD connected to laptop via USB 3.0, have everything moved from server.

Have downloaded File Server 16.1 and began the install, but when attempting the partitioning, get message:

If you wish to use this device for partitioning please remove its LVM data first.

Tried several things, unable to remove LVM data, any help appreciated.

Bob (rzwemer)

P.S. have loaded both CORE & WordPress on Rpi 4 (2GB & 8GB) comes up, can access via browser, new stuff looks better than the old, will do more checking and let you know.

Jeremy Davis's picture

It should be happy to just wipe the disk?! Regardless of LVM data or not. It sounds like perhaps there's a race condition and it's loading the LVM volumes before the partitioner starts and the partitioner doesn't know how to handle that?

You don't explicitly say, but from the way you're speaking, I'll assume that you are installing to bare metal hardware. If I'm wrong and it's a VM, just dump the disk and create a fresh one.

One way to deal with that would be using the "live" functionality of our ISO and just nuke the disk.

To do that, I suggest first physically disconnecting any drives that you don't want wiped. Then boot from the TurnKey ISO and select the live boot option. It will boot up and run through the firstboot process like a clean install does. Then it will load confconsole and you are good to log in. If it's plugged into the network still, then personally I'd ssh in from another PC (better user experience IMO) but were only going to run one command, so you can use the terminal directly if you're happy with that.

Then just manually nuke the disk (assuming /dev/sda):

# this will destroy all data on /dev/sda - i.e. the first disk that your PC will try to boot from
dd if=/dev/urandom of=/dev/sdc bs=1M count=2

Actually contrary to the warning, it will just delete the first 2 MB of the disk's data, but seeing as that's where all the info about the partitions is stored, for all intents and purposes, it's nuked the data and the disk is "clean".

Then just reboot and retry installing. Hopefully it should "just work" now.

Great to hear that the RPi builds are going well for you so far.

rzwemer's picture

Jeremy,

Was offline for the weekend, my one and only grandchild, turned 6, we celebrated this weekend.

Yes it is bare metal hardware, when my 2 Acer Easystore H340's running Windows Home Server were no longer supported by Microsoft, I went looking for an Open Source solution, and found Turnkey File Server, which worked flawlessly for approx. 7 years, therefore the attitude of "If it ain't broke don't fix it"

Thanks for the response, it worked to clean the LVM data, but after rebooting and trying a fresh install am having issues using my cleaned disks.

Following are the results of executing fdisk -l, then checking Partitions on Local Disks with Webmin:

======================================================================================
======================================================================================
fdisk -l
======================================================================================

Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: WDC WD10EAVS-00D
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x18b1e278

Device     Boot      Start        End    Sectors   Size Id Type
/dev/sda1  *          2048 1949351935 1949349888 929.5G 83 Linux
/dev/sda2       1949353982 1953523711    4169730     2G  5 Extended
/dev/sda5       1949353984 1953523711    4169728     2G 82 Linux swap / Solaris
======================================================================================

Disk /dev/sdc: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: Hitachi HDS5C302
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x030f35a3
======================================================================================

Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: WDC WD10EADS-00M
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x08672650

Device     Boot Start    End Sectors  Size Id Type
/dev/sdb1        2048 121600  119553 58.4M 83 Linux
======================================================================================

Disk /dev/sdd: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: Hitachi HDS5C302
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x9230575f
======================================================================================

Disk /dev/sde: 245 MiB, 256901120 bytes, 501760 sectors
Disk model: USB DISK        
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc001a7f4

======================================================================================
======================================================================================
Webmin   Partitions on Local Disks
======================================================================================

Device     Boot Start    End Sectors   Size Id Type
/dev/sde1  *       63 481949  481887 235.3M  7 HPFS/NTFS/exFAT

Partitions on Local Disks
Using fdisk
Disk name    Total size    Make and model    Partitions    Actions
SATA device A     931.51 GiB     ATA WDC WD10EAVS-00D     3     

SATA device B     93.12 GiB     ATA WDC WD10EADS-00M     1     

SATA device C     1.81 TiB     ATA Hitachi HDS5C302     0     

SATA device D     1.81 TiB     ATA Hitachi HDS5C302     0     

SCSI device E     243.17 MiB     SMI USB DISK     1     

======================================================================================
Edit Disk Partitions
SATA device A
Disk size: 931.51 GiB, Make and model: ATA WDC WD10EAVS-00D, Cylinders: 121602, Partition table format: DOS.

Number    Type    Size    Start    End    Used by
1     Linux     929.52 GiB     1     121342     /
2     Extended     1.99 GiB     121342     121602     
5     Linux swap     1.99 GiB     121342     121602     Virtual memory

======================================================================================
Edit Disk Partitions
SATA device B
Disk size: 93.12 GiB, Make and model: ATA WDC WD10EADS-00M, Cylinders: 121601, Partition table format: DOS.

Number    Type    Size    Start    End    Used by
1     Linux     58.81 MiB     2     76     

======================================================================================
Edit Disk Partitions
SATA device C
Disk size: 1.81 TiB, Make and model: ATA Hitachi HDS5C302, Cylinders: 243201, Partition table format: DOS.

This disk has no partitions yet.

======================================================================================
Edit Disk Partitions
SATA device D
Disk size: 1.81 TiB, Make and model: ATA Hitachi HDS5C302, Cylinders: 243201, Partition table format: DOS.

This disk has no partitions yet.

======================================================================================
Edit Disk Partitions
SCSI device E
Disk size: 243.17 MiB, Make and model: SMI USB DISK , Cylinders: 31, Partition table format: DOS.


Number    Type    Size    Start    End    Used by
1     NTFS     235.32 MiB     1     30     

======================================================================================
======================================================================================

Logical Volume Management
LVM version 2.03.02

No volume groups were found on your system.

======================================================================================

Regaining familiarity and appreciation for Linux.

Bob (rzwemer)

Jeremy Davis's picture

I think a birthday party, especially a 6th birthday for your only grandchild is a fine reason to take a weekend off. :)

But back to your issue, I'm not really clear exactly what you are trying to do.

From your output I see 5 disks:

SATA device A     931.51 GiB     ATA WDC WD10EAVS-00D
SATA device B     93.12 GiB     ATA WDC WD10EADS-00M
SATA device C     1.81 TiB     ATA Hitachi HDS5C302
SATA device D     1.81 TiB     ATA Hitachi HDS5C302
SCSI device E     243.17 MiB     SMI USB DISK

First, I assume that some have data on them that you'd like to keep? I'm also guessing that 'E' is your USB installer? Although,actualy, probably not!? (It looks like it's formatted with NTFS which doesn't seem right for our installer?!).

I must admit too that in my ten years with TurnKey, I haven't ever actually done a bare metal install. I run a Proxmox VE hypervisor and all my TurnKey installs are either LXC containers or KVM vms. I'm a big fan of LVM. If you aren't familiar with LVM, it is somewhat like software RAID - it's not really, but from a functional perspective it's a fair analogy. It allows filesystems to span multiple disks and/or can also mirror data across disks/partitions. Another cool feature is that it allows you to take "snapshots" of the filesystem. You shouldn't consider these snapshots backups, but they are a super handy way to roll back if/when things go wrong (e.g. OS upgrade on bare metal).

So I'm not really sure what the issue is or exactly what you are trying to do.

Personally, I'd just decide which disk you want to use as your root volume (i.e. '/') and physically disconnect all others. I'd then install TurnKey and accept the installer defaults (guided partitioning with LVM - only use 90% - 10% free for snapshots) and just let it do what it wants to do. Once that's done, plug the other drives back in, reboot and go through the firstboot inithooks. Once that's done, then format and/or mount the other drives as you wish (making it permanent by adding /etc/fstab entries). How you want to organise your disks and filesystem will depend on how much space you want/need and how much you trust those disks.

I'm more than happy to elaborate on those steps. if/when needed.

If you go with LVM, then you could just add them to the 'root' logical volume and you'd have just one big '/'. Or you could keep them separate (with or without LVM) and use them for different purposes. Or a "mix and match" combo of both approaches.

Personally, I don't trust hard disks. Especially cheaper and/or older disks. I've had lots of disks fail, so for anything that I plan to keep, but too big to justify the expense of offsite backup, only gets stored on a WD Black spinney drive or Samsung SSD (plus mirrored or copied somewhere else local too). I have a particular distaste for Hitachi disks. It's likely unfair, but I had 2 fail on me within weeks of each other with no warning and I lost a ton of data. :(

So whilst I always use LVM, I often don't use it's value by default (I use it more to allow for future options and/or flexibility). For example, I have replaced a spinney server hard drive with an SSD with zero downtime! :) (FWIW, I added the SSD to the LVM, configured it as a mirror of the HDD, then removed the HDD from the mirror and then the LVM and finally the physical machine - no reboot required!).

Anyway, I'm rambling a bit... But beyond LVM, if you want follow my lead, then you'll also need to checkout "bind mounts". My "storage" disks also use LVM but are both individual standalone volumes, using bind mounts to mount one to /srv/storage and the other to a subdirectory of /srv/storage/. Bind mounts allow you to mount an existing filesystem in multiple places.

If you keep in mind that under the hood, TurnKey is Debian (v16.x = Debian 10/Buster - currently 'oldstable'), then you'll find lots of info online (not that I mind answering your questions - I don't - but you likely don't need to wait until I share my feedback).

Alternatively, if you have a specific plan or goal re the usage of your disks and would like some practical advice on getting there, please share the plan.

rzwemer's picture

Jeremy,

Thanks for all the help and advice on this matter, initially when the problem began I was frustrated, now I'm looking at this as yet another opportunity to refresh my understanding of things "under the hood".

First, there is nothing on any of the drives: SATA A, B, C or D, everything has been copied to an 8TB external drive, SCSI device E SMI USB DISK is, best I can figure nand flash memory on the motherboard.

Second, curious about Proxmox VE hypervisor, went to their site and had a look, but don't think it's for my needs now.

Third, removed drives SATA B, C and D installed TurnKey with defaults (guided partitioning with LVM), turned it off, put SATA B, C and D back rebooted don't think it went through firstboot inithooks.

Fourth, both "really old servers" were set up with LVM for the entire set of drives, so each server appeared as a single large drive.

Fifth, I don't trust hard drives either, got me thinking about and researching possible archival means such as using a Blu-ray burner that supports BD XL and M-Disc, back up to BD-R HTL or M-Disc BD-R/DVD+R discs, need to do more poking around about this.

Sixth, at this point I'd like to get this server back up with LVM and SATA A, B, C and D as a single logical volume as it was originally. That said, I would look at some alternative archival/backup methods and would hopefully not return to my old habit of "If it ain't broke don't fix it".

Bob (rzwemer)

P.S. My 6 yr old grandkid spends Mon, Tue and Thurs with us doing remote Kindergarten, so I've been distracted, but Wed and Fri can get stuff done.

Jeremy Davis's picture

For what it's worth. On my server, I have a 60 GB LV (logical volume) on an SSD for Proxmox OS root, a ~620GB LV (spanning remainder of sda and sdb - which is another SSD) for VM root volumes. My other 3 disks are all single disk LVs; a 1TB WD black and 2x 3TB WD red. The black is used for in box backups of important stuff and the 2 reds are file storage for my fileserver (which runs as a VM).

Personally, I love it. It means I only ever need to muck around with bare metal once every few years (when I upgrade Proxmox). My fileserver is an LXC container (with it's OS on an SSD) so runs like it was on bare metal (relative to the resources it's allocated). The 3 spinney disk LVs are mounted into the container within /srv (/srv/backups /srv/storage /srv/storage/video). But I get the flexibility of being able to bring other LXC containers or (KVM) full VMs up or down anytime (which I do a lot because of TurnKey). When I want to upgrade my fileserver, I just launch a new container, transfer across any config I want from the old server (not the files though) then swap the storage disks to the new server. I.e. I can set up the new server before tearing down the old.

Anyway, I digress (although perhaps it's interesting to hear how others do it?).

I'm not sure if you're waiting for me (sorry if you are) but in case you are, and/or someone else comes across this, here's how I'd go form where you are/were.

Double check which disk is the TurnKey install:

root@tkldev6 ~# pvs
  PV         VG      Fmt  Attr PSize   PFree 
  /dev/sda1  turnkey lvm2 a--  

So mine is /dev/sda - yours likely is too, but always best to check. Assuming it is, then you need to choose whether to add partitions to the LVM, or whole disks. From a technical perspective it doesn't matter. You could clear the partition table via the dd command above and then add each disk unpartitioned. Or you can repartition them (e.g. via fdisk) so they are a single partition that takes up the whole disk. Obviously if they are already partitioned like that, you can leave them as is.

Either way, add your other disks (e.g. /dev/sdc) as "physical volumes" (PVs):

# add whole unpartitioned disks and/or partitions:
pvcreate  /dev/sdb1 /dev/sdc #[...]

Then add the PVs to the volume group (VG). If you used defaults, your VG will be called 'turnkey'. You can double check like this:

root@tkldev6 ~# vgs
  VG      #PV #LV #SN Attr   VSize   VFree
  turnkey   1   2   0 wz--n- 

Add the new PVs to the 'turnkey' VG, e.g.:

vgextend turnkey /dev/sdb1  /dev/sdc #[...]

Then add the free space to the LV. By default, the root volume should be called 'root'. Double check like this:

root@tkldev6 ~# lvs
  LV     VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root   turnkey -wi-ao---- 

I suggest keeping 10% free for snapshots and future flexibility, but that's your call. If you do wish to follow my lead, you can roughly work it out how much new space to add via this formula:

VolFree - (VolSize x 10%)

To do the calculation in the CLI, install 'bc':

apt update ; apt install -y bc

Now check 'vgs' and calculate the space to add to your LV. In my case:

root@tkldev6 ~# vgs                                                  
  VG      #PV #LV #SN Attr   VSize   VFree  
  turnkey   3   2   0 wz--n- 

So I'll add that to the LV:

lvextend -L +28.81g /dev/turnkey/root

You should now have roughly 10% free in your VG. E.g. in mine:

root@tkldev6 ~# vgs
  VG      #PV #LV #SN Attr   VSize   VFree
  turnkey   3   2   0 wz--n- 

Near enough! :) Now extend the filesystem:

resize2fs /dev/turnkey/root

The size of the root LV and the size of '/' should be the same:

root@tkldev6 ~# lvs
  LV     VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root   turnkey -wi-ao---- 55.61g 
  swap_1 turnkey -wi-ao----  55G   19G   34G  36% /

Yep. Done deal... :)

rzwemer's picture

Jeremy,

Thanks for the tutorial, I'm getting more comfortable at the CLI again, been awhile.

To begin with, was able to partition using the installation partitioner, then using Webmin add physical partitions to turnkey, thus:

root@fileserver ~# pvs
  PV         VG      Fmt  Attr PSize    PFree                                                                                                                        
  /dev/sda1  turnkey lvm2 a--  <931.51g   93.15g                                                                                                                     
  /dev/sdb1  turnkey lvm2 a--  <929.52g <929.52g                                                                                                                     
  /dev/sdb5  turnkey lvm2 a--     1.98g    1.98g                                                                                                                     
  /dev/sdc1  turnkey lvm2 a--    <1.82t   <1.82t                                                                                                                     
  /dev/sdc5  turnkey lvm2 a--    <1.99g   <1.99g                                                                                                                     
  /dev/sdd1  turnkey lvm2 a--    <1.82t   <1.82t                                                                                                                     
  /dev/sdd5  turnkey lvm2 a--    <1.99g   <1.99g                                                                                                                     
  /dev/sde1          lvm2 ---  <235.30m <235.30m                                                                                                                                                                                                                                          
root@fileserver ~# vgs
  VG      #PV #LV #SN Attr   VSize  VFree                                                                                                                            
  turnkey   7   2   0 wz--n- <5.46t <4.64t                                                                                                                           
root@fileserver ~# lvs
  LV     VG      Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert                                                                             
  root   turnkey -wi-ao---- <836.37g                                                                                                                                 
  swap_1 turnkey -wi-ao----   <1.99g

Then after reading your reply, via Webmin removed them thus:

root@fileserver ~# pvs
  PV         VG      Fmt  Attr PSize    PFree                                                                                                                        
  /dev/sda1  turnkey lvm2 a--  <931.51g   93.15g                                                                                                                     
  /dev/sdb1          lvm2 ---   929.52g  929.52g                                                                                                                     
  /dev/sdb5          lvm2 ---    <1.99g   <1.99g                                                                                                                     
  /dev/sdc1          lvm2 ---    <1.82t   <1.82t                                                                                                                     
  /dev/sdc5          lvm2 ---    <1.99g   <1.99g                                                                                                                     
  /dev/sdd1          lvm2 ---    <1.82t   <1.82t                                                                                                                     
  /dev/sdd5          lvm2 ---    <1.99g   <1.99g                                                                                                                     
  /dev/sde1          lvm2 ---  <235.30m <235.30m                                                                                                                     
root@fileserver ~# vgs
  VG      #PV #LV #SN Attr   VSize    VFree                                                                                                                          
  turnkey   1   2   0 wz--n- <931.51g 93.15g                                                                                                                         
root@fileserver ~# lvs
  LV     VG      Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert                                                                             
  root   turnkey -wi-ao---- <836.37g                                                                                                                                 
  swap_1 turnkey -wi-ao----   <1.99g

Next I followed your lead and thus:

root@fileserver ~# pvcreate /dev/sdb1
  Physical volume "/dev/sdb1" successfully created.                                                                                                                  
root@fileserver ~# pvcreate /dev/sdb5
  Physical volume "/dev/sdb5" successfully created.                                                                                                                  
root@fileserver ~# pvcreate /dev/sdc1                                                                                                                                
  Physical volume "/dev/sdc1" successfully created.                                                                                                                  
root@fileserver ~# pvcreate /dev/sdc5                                                                                                                                
  Physical volume "/dev/sdc5" successfully created.                                                                                                                  
root@fileserver ~# pvcreate /dev/sdd1                                                                                                                                
  Physical volume "/dev/sdd1" successfully created.                                                                                                                  
root@fileserver ~# pvcreate /dev/sdd5                                                                                                                                
  Physical volume "/dev/sdd5" successfully created.                                                                                                                  
root@fileserver ~# vgextend turnkey /dev/sdb1
  Volume group "turnkey" successfully extended                                                                                                                       
root@fileserver ~# vgextend turnkey /dev/sdb5                                                                                                                        
  Volume group "turnkey" successfully extended                                                                                                                       
root@fileserver ~# vgextend turnkey /dev/sdc1                                                                                                                        
  Volume group "turnkey" successfully extended                                                                                                                       
root@fileserver ~# vgextend turnkey /dev/sdc5                                                                                                                        
  Volume group "turnkey" successfully extended                                                                                                                       
root@fileserver ~# vgextend turnkey /dev/sdd1                                                                                                                        
  Volume group "turnkey" successfully extended                                                                                                                       
root@fileserver ~# vgextend turnkey /dev/sdd5                                                                                                                        
  Volume group "turnkey" successfully extended                                                                                                                       
root@fileserver ~# vgs                                                                                                                                               
  VG      #PV #LV #SN Attr   VSize  VFree                                                                                                                            
  turnkey   7   2   0 wz--n- <5.46t <4.64t                                                                                                                           
root@fileserver ~# lvs
  LV     VG      Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert                                                                             
  root   turnkey -wi-ao---- <836.37g                                                                                                                                 
  swap_1 turnkey -wi-ao----   <1.99g                                                                                                                                 
root@fileserver ~#lvextend -L 4t /dev/turnkey/root                                                                                                                   
  Size of logical volume turnkey/root changed from <836.37 GiB (214110 extents) to 4.00 TiB (1048576 extents).                                                       
  Logical volume turnkey/root successfully resized.                                                                                                                  
root@fileserver ~# vgs                                                                                                                                               
  VG      #PV #LV #SN Attr   VSize  VFree                                                                                                                            
  turnkey   7   2   0 wz--n- <5.46t <1.46t                                                                                                                           
root@fileserver ~# lvextend -L +1t /dev/turnkey/root                                                                                                                 
  Size of logical volume turnkey/root changed from 4.00 TiB (1048576 extents) to 5.00 TiB (1310720 extents).                                                         
  Logical volume turnkey/root successfully resized.                                                                                                                  
root@fileserver ~# vgs                                                                                                                                               
  VG      #PV #LV #SN Attr   VSize  VFree                                                                                                                            
  turnkey   7   2   0 wz--n- <5.46t <467.05g                                                                                                                         
root@fileserver ~# resize2fs /dev/turnkey/root
resize2fs 1.44.5 (15-Dec-2018)                                                                                                                                       
Filesystem at /dev/turnkey/root is mounted on /; on-line resizing required                                                                                           
old_desc_blocks = 105, new_desc_blocks = 640                                                                                                                         
The filesystem on /dev/turnkey/root is now 1342177280 (4k) blocks long.                                                                                              
                                                                                                                                                                     
root@fileserver ~# lvs                                                                                                                                               
  LV     VG      Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert                                                                               
  root   turnkey -wi-ao---- 5.00t                                                                                                                                   
  swap_1 turnkey -wi-ao---- <1.99g                                                                                                                                   
root@fileserver ~# df -h /
Filesystem                Size  Used Avail Use% Mounted on                                                                                                           
/dev/mapper/turnkey-root  5.0T  2.4G  4.8T   1% /

And you are correct:

Yep. Done deal... :)

My next adventure will be to follow this process to upgrade my other bare metal Turnkey File Server, maybe I'll try again when v17.0 arrives.

Bob (rzwemer)

P.S. Again, Jeremy can't thank you enough for the continued support with this matter.

rzwemer's picture

Jeremy,

Forgot to mention, will continue to poke around with both CORE & WordPress on Rpi 4 (2GB & 8GB).

Bob (rzwemer)

Add new comment