I messed up. How do I fix it?

TBH, I'd never heard of Softether until now, so other than the really brief read I just had, I'm not very familiar with it, but it does sound quite cool.

Using VPN behind port 53 shouldn't be any issue, but I'm not surprised that setting it up to listen on ports 25 and 443 is causing issues. As you possibly know, port 25 is used by SMTP and port 443 is used by Apache (HTTPS). By default only one application can listen on any particular port.

I can certainly understand why Softether may suggest or recommend you use these particular ports for VPN. As they are "standard" ports for other protocols, they are often unblocked by corporate firewalls. Although OTOH, port 25 is often blocked by ISPs

Personally, I'd recommend that you start using ports that aren't required for other protocols. Then use those for your VPN traffic. Or you could stop those services from listening on those ports (as it sounds like you tried doing).

If you really need to use those ports for both VPN and SMTP, then you'll need to install some sort of protocol multiplexer to listen on those ports (e.g. sslh). Then depending on the traffic it gets, it will forward to the relevant application.

As for what has gone wrong with your current set up. TBH, I'm not really sure, but I'd recommend that you inspect the logs and double check that all the services are running. Both Webmin and Webshell (aka shellinabox - that's the service name) are behind stunnel, so also check that stunnel is running. E.g. something like this:

service webmin status
service shellinabox status
service stunnel4 status

service webmin start
service shellinabox start
service stunnel4 start

If any of them fail to start, then hopefully the status messages will give you a hint of what the issue may be and/or what logs to check. Logs can generally be found in /var/log. If you are running this as a "proper" VM (i.e. not a container) then you can also leverage SystemD's journalctl tool to inspect SystemD's binary logs.

Personally, in situations like this, I would disable the firewall until you have resolved the issue. Otherwise it's really hard to work out whether it's a firewall issue, or something else. Then once you have everything working as it should, then re-enable the firewall. If that breaks things, then you know that you'll need to tweak your firewall rules.

Adminer runs on Apache and TBH, the error that you are noting sounds weird. Perhaps MySQL isn't running either? (Try above commands, but use mysql as the service name). Alternatively, it could just be a cookie issue. Perhaps try logging in from a different web browser, or use "private" or "incognito" mode (depending on what browser you're using, it's called different things).

Another thing worth keeping in mind is that TurnKey comes with etckeeper preinstalled. That keeps the whole /etc directory (where most of the config settings are) under version control (git). So you should even be able to checkout previous versions of your config files if you wish.

Also, out of interest, if you have a current backup, why not just restore this server? Or did you want to try to fix this one?

I guess it is possible that Softether work around that by providing normal HTTP headers to it's traffic. If so, then it should be able to work as you hope. If so, it should be clearly documented and may also be limited to a specific webserver (I'd consult the Softether docs).

Even if it does, IMO that would be a clear example of feature creep (i.e. providing additional features beyond your intended functionality). Personally I tend to avoid software that has feature creep where ever possible. I think the old school Unix philosophy, "do one thing and do it well" is how things should be done. And in this case, IMO what you are trying to achieve would be best accomplished using the modular approach. I.e. a multiplexer such as sslh. That reads the headers and would hand HTTP/HTTPS traffic to Apache and VPN traffic to Softether.

I still think that using sslh (e.g. here's a post about using port 443 for both SSH and HTTPS) would be a great way to achieve your ends. Although AFAIK, VPN would work with either domain.

Using the LXC appliance may be another way to work around the issue you are experiencing. I don't recall how it's set up in relation to domain names, but even if it doesn't support named based redirection OOTB, you could bend it to do that I'm sure. And that way, you could ensure that VPN is only available via that particular url.

Thank you the detailed response Jeremy.

Maybe I'm misunderstanding how Apache is able to determine the hostname being used to access a port and redirect to the appropriate resource. Does this only happen with HTTP requests? I thought it could apply to all attempts to connect to that specific IP:port combination so if there's a VPN connection attempt--for example--the request would be passed on to the VPN handler.

What I had envisioned:

https request sent to www.domain.com on port 443 is redirected to web handler.
VPN request sent to vpn.domain.com (same IP) on port 443 is redirected to VPN handler.

I know apache is able to distinguish between which hostname is used to access a resource. It's unclear if it's limited to HTTP, or can apply to multiple protocols.

Running a tklbam-restore to this server failed after about 45 minutes. Looking at the disk usage, there simply wasn't enough space to download the duplicity files necessary to return to 3 days ago. I ultimately gave up last night trying to figure out exactly how to undo what I did wrong, and instead opted for a clean install.

So I set up a virtualbox instance on my computer with 3x the disk space of the failed VPS, and set up the restore to run there. That process finished a little while ago. I made sure all the necessary services are running on the virtualbox instance, and then I went ahead and ran the reinstall process on my VPS host. Once that finished, I started an rsync job to copy all the contents of the virualbox instance to the VPS instance.

I'll find out in another 13 hours (ADSL line--last resort) if this is a success.

I figured out the adminer issue. The login page doesn't seem to be behind apache, but the login POST request fails in the same way as the webmin login page and the WebShell page. Basically a 403 error with something along the lines of connection refused.

Thank you for pointing out etckeeper. I'll keep that in mind should this happen again.

Out of curiosity, would it make sense to use the LXC (/lxc)?

The LAMP stack instance would be in one container with ports forwarded from the nginx master so specific domains will forward to the LAMP stack, while a different set of domains--still using the same ip and port--will forward to other containers.