K. Callis's picture

I have been pulling my hair out on the fact that I have not been able to access webmin or even the webshell. I looked at the message:

root@fs-01 ~# journalctl -b -u webmin.service -u stunnel4@webmin.service

-- Logs begin at Tue 2022-08-30 03:35:55 CDT, end at Tue 2022-08-30 03:48:01 CDT. --
Aug 30 03:35:57 fs-01 systemd[1]: Starting Universal SSL tunnel for network daemons (webmin)...
Aug 30 03:35:59 fs-01 stunnel[235]: LOG5[ui]: stunnel 5.50 on x86_64-pc-linux-gnu platform
Aug 30 03:35:59 fs-01 stunnel[235]: LOG5[ui]: Compiled with OpenSSL 1.1.1b  26 Feb 2019
Aug 30 03:35:59 fs-01 stunnel[235]: LOG5[ui]: Running  with OpenSSL 1.1.1n  15 Mar 2022
Aug 30 03:35:59 fs-01 stunnel[235]: LOG5[ui]: Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,O
Aug 30 03:35:59 fs-01 stunnel[235]: LOG5[ui]: Reading configuration from file /etc/stunnel/webmin.conf
Aug 30 03:35:59 fs-01 stunnel[235]: LOG5[ui]: UTF-8 byte order mark not detected
Aug 30 03:35:59 fs-01 stunnel[235]: LOG5[ui]: FIPS mode disabled
Aug 30 03:35:59 fs-01 stunnel[235]: LOG5[ui]: Configuration successful
Aug 30 03:35:59 fs-01 stunnel[235]: LOG5[ui]: Binding service [webmin] to :::12321: Address already in use
Aug 30 03:35:59 fs-01 stunnel[235]: LOG5[ui]: Switched to chroot directory: /var/lib/stunnel4/
Aug 30 03:35:59 fs-01 stunnel[306]: LOG3[main]: Cannot create pid file /webmin.pid
Aug 30 03:35:59 fs-01 systemd[1]: stunnel4@webmin.service: Can't open PID file /var/lib/stunnel4/webmin.pid
Aug 30 03:35:59 fs-01 stunnel[306]: LOG3[main]: create: Permission denied (13)
Aug 30 03:45:57 fs-01 systemd[1]: stunnel4@webmin.service: Start operation timed out. Terminating.
Aug 30 03:45:57 fs-01 systemd[1]: stunnel4@webmin.service: Failed with result 'timeout'.
Aug 30 03:45:57 fs-01 systemd[1]: Failed to start Universal SSL tunnel for network daemons (webmin).
Aug 30 03:45:57 fs-01 systemd[1]: Dependency failed for Webmin Web based Admin UI.
Aug 30 03:45:57 fs-01 systemd[1]: webmin.service: Job webmin.service/start failed with result 'dependency'.
Aug 30 03:45:57 fs-01 systemd[1]: stunnel4@webmin.service: Consumed 16ms CPU time.
Aug 30 03:45:58 fs-01 systemd[1]: stunnel4@webmin.service: Service RestartSec=1s expired, scheduling restar
Aug 30 03:45:58 fs-01 systemd[1]: stunnel4@webmin.service: Scheduled restart job, restart counter is at 1.
Aug 30 03:45:58 fs-01 systemd[1]: Stopped Universal SSL tunnel for network daemons (webmin).

and

 

root@fs-01 ~# service webmin status
* webmin.service - Webmin Web based Admin UI
   Loaded: loaded (/lib/systemd/system/webmin.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/webmin.service.d
           `-override.conf
   Active: inactive (dead)

Aug 30 03:45:57 fs-01 systemd[1]: Dependency failed for Webmin Web based Admin UI.
Aug 30 03:45:57 fs-01 systemd[1]: webmin.service: Job webmin.service/start failed with result 'dependency'.

 

With the exception updating the system I have not even had a chance to configure the system, so why am I having this issue?

Forum: 
Jeremy Davis's picture

It looks like stunnel can't write it's "pid" file?!:

Aug 30 03:35:59 fs-01 stunnel[306]: LOG3[main]: Cannot create pid file /webmin.pid
Aug 30 03:35:59 fs-01 systemd[1]: stunnel4@webmin.service: Can't open PID file /var/lib/stunnel4/webmin.pid
Aug 30 03:35:59 fs-01 stunnel[306]: LOG3[main]: create: Permission denied (13)

The message suggests that the process doesn't have permission. If it is indeed permissions, then I'm not sure how how that happened? But this should fix it (comment before each line explains what it does):

# ensure all relevant services are stopped
systemctl stop webmin shellinabox stunnel4@webmin stunnel4@shellinabox
# ensure that the path exists
mkdir -p /var/lib/stunnel4
# remove any pid files that may already be present
rm -f /var/lib/stunnel4/*.pid
# ensure correct ownership
chown stunnel4:stunnel4 /var/lib/stunnel4
# ensure correct permissions
chmod 0755 /var/lib/stunnel4

Now start webmin and webshell (aka shellinabox) services:

systemctl start webmin shellinabox

Check that all 4 services are now running (the stunnel services should be autostarted by Webmin and Webshell services):

# return just the first few lines of each services' systemctl output
systemctl status webmin shellinabox stunnel4@webmin stunnel4@shellinabox | grep '^*' -A5
* webmin.service - Webmin Web based Admin UI
     Loaded: loaded (/lib/systemd/system/webmin.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/webmin.service.d
             `-override.conf
     Active: active (running) since Tue 2022-08-30 20:47:44 UTC; 6min ago
    Process: 269013 ExecStart=/usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf (code=exited, status=0/SUCCESS)
--
* shellinabox.service - Shell In A Box Daemon (aka WebShell)
     Loaded: loaded (/etc/init.d/shellinabox; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-08-30 20:47:40 UTC; 6min ago
    Process: 269011 ExecStart=/etc/init.d/shellinabox start (code=exited, status=0/SUCCESS)
      Tasks: 2 (limit: 9507)
     Memory: 1016.0K
--
* stunnel4@webmin.service - Universal SSL tunnel for network daemons (webmin)
     Loaded: loaded (/lib/systemd/system/stunnel4@.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-08-30 20:47:40 UTC; 6min ago
    Process: 269003 ExecStart=/usr/bin/stunnel4 /etc/stunnel/webmin.conf (code=exited, status=0/SUCCESS)
   Main PID: 269009 (stunnel4)
      Tasks: 2 (limit: 9507)
--
* stunnel4@shellinabox.service - Universal SSL tunnel for network daemons (shellinabox)
     Loaded: loaded (/lib/systemd/system/stunnel4@.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-08-30 20:47:40 UTC; 6min ago
    Process: 269004 ExecStart=/usr/bin/stunnel4 /etc/stunnel/shellinabox.conf (code=exited, status=0/SUCCESS)
   Main PID: 269007 (stunnel4)
      Tasks: 2 (limit: 9507)

Note that all 4 state "Active: active (running)". That suggests all is well.

If that's not it. The only other thing that occurs to me as a possibility is you've run out of free space. Check that like this:

df -h /

PID files are usually tiny, so for it to be caused by lack of free space, then I would expect the drive to be reporting 100% use. If it's less than that, I suggest also checking free inodes (data structure for file metadata). It's unlikely in my experience, but worth checking:

df -i /

If none of that helps and/or you are still experiencing issues, please share the output of the df commands above, as well as the last 40 lines of stunnel@webmin's journal:

journalctl -u stunnel4@webmin | tail -40

It's probably also worth sharing how the pid directory looks now too:

ls -la /var/lib/stunnel4
K. Callis's picture

Following the posting, it seemed that all is well. The only issue is that I have now is that when I try to connect, it get a message saying:

 

Error - Document follows

This web server is running in SSL mode. Try the URL https://localhost:10000/ instead.

 

I have seen posting where there was some issue that either /etc/webmin/miniserv.conf or /etc/stunnel/webmin.conf needed to be changed. If I connect to https://192.168.30.20:10000, I do get to webmin login page, but that would seem that something is wrong with my configuration.

Jeremy Davis's picture

Well I'm glad to hear that you managed to resolve the other issue.

By default, we hide Webmin and Webshell behind Stunnel (aka stunnel4). Stunnel is a super lightweight SSL/TLS proxying tool. Essentially it takes care of the HTTPS, then hands to Webmin (or Webshell). Part of the reason for that is so we can use a single default SSL/TLS cert for all HTTPS traffic on your server. There was also a historical context, but strictly speaking it's not required (at least not for Webmin).

So you have some choices:

  • Reconfigure things so they are back to the TurnKey default (behind stunnel); or
  • Ensure that Webmin doesn't use Stunnel and serves itself publicly; either via port 10000 (default Webmin port) or 12321 (TurnKey Webmin default port).

If you'd like to use the TurnKey default, then you'll need to turn off SSL/TLS (HTTPS) in Webmin. That is because Stunnel decrypts the traffic already, so only sends vanilla HTTP. As you have Webmin configured to only accept HTTPS, it doesn't understand the traffic it is receiving.

You'll also want to ensure that Webmin is only listening on localhost. The fact that you can connect to Webmin from your PC via port 10000 means that Webmin is listening publicly. That's not what you want if you're using our default config (i.e. behind Stunnel) - especially once you disable HTTPS!

Both of those config changes should be achievable by editing Webmin mini-server config file (/etc/webmin/miniserv.conf). First be sure to stop Webmin (and it's stunnel counterpart).

systemctl stop webmin stunnel4@webmin

Here are the settings you'll want (if they already exist, ensure that the values match, if they don't already exist, add them on a new line):

ssl=
no_ssl2=1
no_ssl3=1
no_tls1=1
no_tls1_1=1
bind=127.0.0.1

Also please be aware that it's best not to edit Linux files with Windows tools (such as Notepad) as that will often cause issues (Windows uses different line end characters by default which makes it hard for some Linux apps to be able to understand). So I recommend just editing the files in place (i.e. via CLI within a SSH session) using tools such as nano (pretty much like Notepad but for Linux CLI) or vim (if you're already a vim user, I don't need to say anymore - other than I suggest installing "proper" vim - TurnKey only includes "vim.tiny" by default). If you're hellbent on using Windows, then I suggest a third party tool which can edit Linux files without mangling them, e.g. Notepad++.

If you use our defaults and want a "proper" HTTPS cert, then please check out our Confconsole tool, particularly the Let's Encrypt integration. That can get you a single CA signed TLS cert that will "just work" for the main webserver, plus Webmin and Webshell.


The other option would be to get Stunnel out of the way and just allow Webmin to serve itself publicly. In that case, you just need to clean up a little config, but basically leave it as it is now. At the least you will at least want to disable Stunnel for Webmin and remove Webmin's custom config that we add (which makes it depend on Stunnel). Do both of those like this:

systemctl disable --now stunnel4@webmin
mv /etc/systemd/system/webmin.service.d /etc/systemd/system/webmin.service.disabled
systemctl daemon-reload
systemctl restart webmin

If you'd rather use the default TurnKey port (12321), then just change these Webmin miniserv.conf values:

port=10000
listen=10000

Instead of '10000', change both of those to whatever port you wish to use (e.g. '12321' as we use by default).

As a final check, see what ports and which interfaces your server is listening on:

netstat -tlnp

On a default TurnKey LAMP based server, it should look something like this:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      620/mariadbd        
tcp        0      0 127.0.0.1:10000         0.0.0.0:*               LISTEN      269038/perl         
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      439/sshd: /usr/sbin 
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1270/master         
tcp        0      0 127.0.0.1:12319         0.0.0.0:*               LISTEN      269036/shellinaboxd 
tcp        0      0 0.0.0.0:12321           0.0.0.0:*               LISTEN      269009/stunnel4     
tcp6       0      0 :::80                   :::*                    LISTEN      445/apache2         
tcp6       0      0 :::22                   :::*                    LISTEN      439/sshd: /usr/sbin 
tcp6       0      0 :::443                  :::*                    LISTEN      445/apache2         
tcp6       0      0 :::12320                :::*                    LISTEN      269007/stunnel4  

In that output, you can see Webmin ('269038/perl ') is listening on '127.0.0.1:10000'; i.e. localhost, port 10000. Stunnel (or at least '269009/stunnel4') is listening publicly on port 12321 (i.e. '0.0.0.0:12321'). I have noticed in my output that one stunnel (actually Webshell's one) is listening on IPv6 (which also implies that it's still also listening via IPv4), but Webmin's only appears to be listening on IPv4? TBH, I have no idea why that is... I may need to do some more investigation there...

K. Callis's picture

Thanks for the assistance! Before I start, the very first thing I pull down after update and upgrade is neovim! :) All is well in the universe as I am able to go to webmin with no issue now. I went with stunnel path, made the appropriate changes, reboot and nothing but joy.

Now I can see if I can get snapraid and mergerfs working and life will be groovy, but that is another day. I am running File Server as an lxc container on proxmox. I was initially running Open Media Vault as a VM, but the resources was high. Now I am using a whopping 124mb of RAM (although my CPU utilization is seemingly high). So I thank you very much on getting this working properly.

Jeremy Davis's picture

Nice! I'm actually still using vim, but a close colleague always laughs at me and asks why I don't use neovim. He says it's exactly the same as vim, except faster and better! So I probably should give it a go! :)

I have no experience with snapraid or mergerfs. A quick google suggests that they are a fairly popular pair!? I'm not sure about snapraid, but my brief reading suggests that mergerfs should definitely be compatible with LXC, although you will need to install it on your Proxmox host before you can use it within an LXC guest. I found a few threads on the Proxmox forums that talk about it, as well as some threads on Reddit.

Anyway, glad to hear that you're up and running.

Add new comment