Security Issues with TurnKey Linux Wordpress

Hi Michel,

Can you elaborate on your security breach. How do you know you were breached. It might help us to advise next steps.

You asked for documentation for non-linux folks...but TKLX is linux...are you running this in a container within another non-linux O/S (Windows / Mac)?

You said WordPress was not compromised...but the database might have been or one of the plug-ins might have been. What makes you think this is a server-level breach?

I have been running a number of TKLX WordPress servers for a number of years with no security breaches. I disable root login and change the admin accounts to 32 character random account names with 64 character random passwords and lock down the ssh port to my ip address only. My WordPress has brute login protection and captchas on all forms. I rename the WordPress account names / passwords to also be long random strings.

When it comes to security you need to assume you will be hacked at some point. Your goal isn't to prevent it but slow it down, detect it, deflect it, minimise the damage and have a recovery strategy. That's all you can do. Everyone can be hacked. A few weeks ago it was Homeland security.

Just a suggestion :-)

Hi Michael,

Thanks for the additional details. Tracking this down is going to require some iterative steps on your part though.

Does your NGFW protect at layer7 (application layer)? If not (most do not) then any intrusion you have, if any, is likely at the application level. The fact that your other servers do not have an issue would also indicate an application level intrusion, likely through your WordPress plugins, if this is an intrusion at all. Your rogue connections could just be SaaS calls from your WordPress plugins.

Next step would be as follows:

1) Is this reproducible (do these rogue links reconnect)?

2) If the rogue connections reoccur is it only on boot or do they reconnect themselves after you disconnect them?

3) What are the IP addresses for the rogue connections? (let's see who is on the other end)

4) You stated that the rogue connections caused "the database to stop loading". Could you clarify this?

Hi,

Sorry for the late response to this.

I say it is with the server itself and not wordpress because I use "Sucuri Security" which performs malware and other security scans on the wordpress files.  And nothing malicous was detected with any of the wordpress files.  Though, when I use the command "netstat -na | grep ESTAB", I saw several HTTP connections to some foriegn server which causes the database to stop loading on the TKLX appliance.  Hence, breaking wordpress.  If I disconnect the server from the Internet then those outbound connections stop and everything is fine.  Only HTTP and HTTPS is allowed to the server.  Plus I have inbound and outbound IPS, AVS, and Malware policies on our NGFW to that server.  My other web servers, which are NOT TKLX appliance don't experience this problem.

So something comproised that server and I have no way of doing any easy-to-use scan on the entire TKLM to detect for any exploits and cleaning it.  Like what you can easily accomplish for any Windows-based system.

I'm trying to find some user-friendly documentation, tools, or resources for performing a security scan on a linux system to locate and remove the exploits.  That is why I have to rebuild the server from scratch and do a restore from backup because I am unable to locate such tools or resources.

Furthermore, if there are any GREAT web pages for hardenining a Wordpress site, that would be great also.

Any help with that would be greatly appreciated!

Michel

Apologies on being so slow to the party...

Unless you have reconfigured Apache to serve additional content then I can't see how it could be anything but a compromised WordPress. By default TurnKey WordPress appliance will only accept HTTP connections on Port 80 (all other connections should only be HTTPS or SSH - although it sounds like you have disabled/blocked that) and the only site available on port 80 is WordPress.

As Tim already hinted, other than via WordPress itself, the most common way that a server is compromised is via a poor root password (attacked via SSH). However if that's what happened then most often your server would become part of a botnet and you would be seeing some really high network traffic. Besides it seems unlikely seeing as you suggest that SSH is blocked.

To answer your specific questions:

1. Security updates - Security updates auto install every night. You can check when it last ran by inspecting /var/log/cron-apt/log. If you want to know more about the packages that were installed then check /var/log/apt/history.log. E.g.:

tail /var/log/cron-apt/log
tail /var/log/apt/history.log

turnkey-install-security-updates

apt-get update && apt-get upgrade

2. I mentioned the cron-apt and apt logs above (for auto-update checking and update installs respectively). Other logs that may be of specific interest are Apache connections: /var/log/apache2/access.log. I assume that WordPress also keep a log although I am not sure whether it's enabled by default. I suspect that it would be somewhere within the WordPress root (/var/www/wordpress). All the system level logs are in /var/log.

3. Under the hood TurnKey is Debian (v14.x = Debian Jessie). There are many anti-malware tools for Linux but TBH I have never used any of them. I have only ever had two servers hacked. And both were because of poor passwords (one many years ago and more recently a test server I forgot to turn off after I was finished).

FWIW the most popular Linux anti-malware tools are ClamAV, Chkrootkit and Rkhunter. All of them should provide install instructions for Debian and I suggest you just follow them (instructions for Ubuntu are probably also close enough in most cases). There are also a number of proprietary/commercial names that you may be familiar with (if you use Windows) however AFAIK they offer no real advantages on Linux above and beyond the free tools. You could also enable the firewall and block the dodgey remote connections. I highly doubt that will solve the issue but may at least reduce the load on your server while you run tests.

To remove the possibility of compromised binaries (if you have indeed had a OS level breach) then a TKLBAM backup restored to a new server should resolve that. Although there is also a risk that you take the malware with you, so I recommend that you run the above tools first and do that as an additional layer of "piece of mind".