You are here
TurnKey 13 critical security issue (Heartbleed / CVE-2014-0160)
Without action, your TurnKey 13 installations may remain vulnerable to the critical Heartbleed OpenSSL attack (DSA-2896-1 CVE-2014-0160). This is not a theoretical attack. A remote exploit already exists in the wild.
TurnKey installations are configured to install security updates automatically. Unfortunately, according to our testing installing the update is not enough.
For the fix to take effect you'll need to either:
1) Reboot your system:
/sbin/reboot
2) Individually restart any service (e.g., webserver, shellinabox, webmin, etc.) that uses the OpenSSL library:
/etc/init.d/apache2 restart
/etc/init.d/nginx restart
/etc/init.d/lighttpd restart
/etc/init.d/shellinabox restart
/etc/init.d/webmin restart
Your private SSL key may already have been stolen
Fixing this vulnerability only protects you from future attacks. An attacker can use this attack to read arbitrary chunks of server memory containing private keys, sensitive login credentials or supposedly encrypted communications. The implications of this are profound.
If an attacker has already exploited heartbleed to to steal your SSL private key she can decrypt all past and future encrypted traffic even after the vulnerability has been fixed.
If this concerns you, you may want to seriously consider issuing new private keys and passwords for all effected services.
Are your servers vulnerable?
Debian Wheezy (stable), OpenSSL 1.0.1e-2+deb7u4 (vulnerable)
Debian Squeeze (oldstable), OpenSSL 0.9.8o-4squeeze14 (not vulnerable)
TurnKey 12.x solutions (based on Debian Squeeze) are NOT vulnerable to Heartbleed, as they shipped with a version OpenSSL that is not affected.
TurnKey 13.x solutions (based on Debian Wheezy) ship with a version of OpenSSL that IS vulnerable.
Additionally, the appropriate steps need to be taken for potentially leaked primary key material, secondary key material, protected content and collateral.
What about new deployments?
New deployments should not be affected as security updates are applied before the services are started.
Testing for vulnerability
You can use either the online or CLI utilities to test for Heartbleed:
Comments
Small window on New Systems may turn into a large window.
Systems which run on static ip's (rather than DHCP) do not get system updates at install time - first boot. It would be great if Networking could be defined before updates are called as it often happens that having installed a new system, then a static ip, Apt-Get updatemay not be configured to auto-run.
Perhaps system updates are automatic but it seems that if not called at first boot these are not always automatic.
Thank you for the notice on this. In terms of "issuing new keys" what would be the best proceedure? Any risk of losing control of ones system in the process?
Installing security updates
Meanwhile you can install security updates manually with the following command:
Re: Installing security updates
Aah, I never knew that command existed. I've always been searching through inithooks to find the file. One thing I'd recommend though is prefixing all of these special commands with turnkey- so they can easily be found by typing turnkey<tab><tab>.
Information is free, knowledge is acquired, but wisdom is earned.
Autocomplete support is a good idea
Another suggestion
Liraz,
Another suggestion would be to link 'turnkey-console' to 'confconsole'. I can remember confconsole now after using TurnKey for a number of years, but new users might find a consistent naming scheme easier to use.
Information is free, knowledge is acquired, but wisdom is earned.
That's not a bad idea John
I just put a new 'issue' on the TKL tracker
Hehe! Nice work Liraz... Too fast for me! :)
Didn't find anything?
I ran that command and nothing was found to install/upgrade. Does that mean I'm not vulnerable? In theory I should be, based on the OP.
You may still need to reboot
To enable usage of the updated components all services that are using OpenSSL need to be restarted. The easiest way to do that is to reboot...
unless of course you are running your server on S3 backed Amazon instance (or some other platform that doesn't support rebooting and keeping your data...). If that's the case you'll need to work out which services need restarting and restart them, or do a TKLBAM backup and restore to a new instance...
Invoking inithooks to regenerate keys
updates at install time - first boot
I just reinstalled the torrentserver and the fileserver on static IP and they both updated automatically.
You should be fine...
When installed on first boot, OpenSSL would have installed before the services that rely on it would have started so you should be fine. Also the self-signed certs would have been regenerated.
SSH keys also at risk?
I'm certainly no expert when it comes to security, but from some of the articles on Heartbleed I gathered that SSH keys are also at risk because they are stored in memory on the server. After all the updates are in place, I plan to regenerate all new keys for users and hosts. I know that most of my virtual hosts are probably safe because they weren't exposed to the Internet but it's as hard to figure out which ones have exposure as it is to change them all.
Information is free, knowledge is acquired, but wisdom is earned.
SSH keys may be at risk, better safe than sorry
OTOH, if SSH (or any other process) doesn't fully wipe out sensitive key material from memory before a sub-process exits it may be possible for the SSH key to eventually find itself in the SSH process's memory space.
It might be overkill to generate the SSH keys but better safe than sorry I guess:
If (purchased)SSL Certs are on Sites, do they need to be rebuilt
Does the OpenSSL system also effect the key generation for Name Based SSL Certs or is this just for the self generated Certs that run by default on WebShell, Webmin and Apache without further configuration of SSL?
Purchased SSL certs will need to be reissued
Confirming update of OpenSSL
Security updates are all applied it appears, but checking openssl version shows it still to be 1.0.1e
Can you confirm that this is correct?
Yes that is correct
As per usual practice in Debian, security patches are backported to the stable version, rather than an updated version being included (which might accidentally break things). That's why they call it Debian stable! :)
FWIW I'll give you a little more info so you can see for yourself (rather than just having to take my word for it...)
If you have a look at the OpenSSL package in Debian Wheezy you will see that the version is '1.0.1e-2+deb7u6'. The 'u6' on the end means that this one is the 6th update since the package version was frozen (when Debian 7 was in the process of moving from 'testing' to 'stable'). If you have a look at the bug reports for OpenSSL on Debian 'stable' firstly you'll see that things have obviously moved pretty fast as this is actually the bug page for '1.0.1e-2+deb7u4' (2 updates ago). But you'll see that "CVE-2014-0160 heartbeat read overrun (heartbleed)" gets a mention right near the top. The fact that the bug number (#743883) is struckout out is a dead giveaway that it has been fixed. You'll also notice that both the bug number and the name are both links. They link to the full bug report log. If you scroll down to message #14 you'll see that on Mon, 7 Apr 2014:
FWIW it looks like it was actually fixed in '1.0.1e-2+deb7u5' so there has actually been another update since Monday when this bug was fixed... As the changelog docs aren't yet available we can't (at least easily) see why they released another new version but they did...
Status of different versions of OpenSSL
What versions of the OpenSSL are affected?
OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable
OpenSSL 1.0.1g is NOT vulnerable
OpenSSL 1.0.0 branch is NOT vulnerable
OpenSSL 0.9.8 branch is NOT vulnerable
Bug was introduced to OpenSSL in December 2011 and has been out in the wild since OpenSSL release 1.0.1 on 14th of March 2012. OpenSSL 1.0.1g released on 7th of April 2014 fixes the bug.
This bug should only affect TKL v13.0
TKL v13.0 (based on Debian Wheezy) has OpenSSL v1.0.1e. But the security bug was patched in 'openssl 1.0.1e-2+deb7u5' (see DSA-2896 and/or CVE-2014-0160).
FWIW the current Wheezy OpenSSL package is 'openssl 1.0.1e-2+deb7u6'.
Also AFAIK (and according to the DSA & CVE I linked to above) it wasn't patched in later versions of Debian until 1.0.1g-1 (patch applied to 1.0.1g in Jessie/testing and Sid/unstable). Not sure what that means for upstream or other platforms though...
Determining Services using OpenSSL
OK... I'll bite. How do I determine which services utilize OpenSSL?
My submission is triggering the spam filter... ugh!
Drew
That's why Alon suggested reboot
Generally the relevant webserver (and perhaps openssh-server) would be the ones to restart. But then Webmin has it's own miniserver (so you'd need to restart that too).
So easiest thing to do is reboot (assuming you're not running on an S3 backed AWS server!!)
Thanks!
Thanks, Jeremy! I just restarted anything that could be restarted. If it couldn't, well, I didn't. It didn't take too long.
Drew
How do I find out what version
How do I find out what version of turnkey my server is running?
Also when listing versions of the OS (Debian), please list NUMBERS not names. I have no idea what wheezy means when webmin lists version numbers of the OS.
I Googled it, too
Google Query: how to determine which version of debian I'm running?
First response took me to a forum and about 2/3 down the page, there was this command:
Here's my server's response:
In Webmin, go Tools > Command Shell:
Drew
Great post Drew! :)
Nice work mate! Sometimes us Linux guys forgot that some people don't know off the top of their head which Toy Story character is associated with which version of Debian! :)
Re: How do I find out what version
Starting with wheezy (7.x) lsb_release -a also displays the Debian release number and name;
but you have to remember the '-a' otherwise you only get the unhelpful first line. I always forget that and think it's broke.
Information is free, knowledge is acquired, but wisdom is earned.
Apps linked to OpenSSL
Older versions of hMailServer (an open source POP3/SMTP/IMAP server for Windows) had a static link to a vulnerable OpenSSL lib. The makers of hMailServer were quick with a notification and an update to address the Heartbleed bug.
But I wonder what other applications have such static links and need updating. Perhaps the SSL modules in PHP and Apache HTTP Server? I've searched the web but find no reports or listings indicating which apps have such issues. Any ideas?
- Michael
Quite likely in Windows
But because of the way that Linux usually does things (i.e. via package management) it is unlikely that any apps (at least ones installed by package management) would have that sort of issue...
List of vendor announcements regarding Heartbleed
The SANS web site has a forum thread listing numerous vendor notification regarding Heartbleed.
- Michael
Thanks Michael
FWIW I signed up and added this blog post/announcement to the list (plus Amazon's...)
Miscellaneous links and comments on Heartbleed
I found another compilation of vendor Heartbleed announcements at this link.
There's no Heartbleed bulletin on the PHP website yet, but at the top of the Windows PHP site, a bulletin says some older versions of PHP 5.5 on Windows use vulnerable versions of OpenSSL DLL's. They provide new Windows PHP binaries as well as new OpenSSL DLLs to address this. It is not clear whether Linux versions of PHP before 5.5.11 are vulnerable.
The common advice is to simply replace the vulnerable OpenSSL 1.0.1 libraries with OpenSSL 1.0.1g versions; however:
1) When the application statically links to the vulnerable OpenSSL libraries, you have to update the application.
2) When the application uses the dynamic OpenSSL 1.0.1 libraries, it may be difficult to compile your own replacement OpenSSL 1.0.1g libraries, and to know that you are using the exact same compiler options as were used by the people who built the application.
- Michael
Similar response to my post above...
Generally in Linux apps are built in a very modular way and use common libraries to fulfill common functions. So it would be highly uncommon for any app like PHP to be a problem on Linux (as it would rely on the OpenSSL package for SSL).
So in the case of Linux as long as software was installed from package management (and often even third party installs) would use the OpenSSL package from the repo. Thus only OpenSSL needs replacing (and the program restarted).
As I also wrote in a comment above, the version in Linux is not always indicative of specific security bugs. Debian Wheezy (aka 7 - the basis of TKL v13.0) still has OpenSSL 1.0.0e - but it is no longer vulnerable (as of package 1.0.0e-2+deb7u5).
Wunderlist Email on Heartbleed (summary at bottom)
Wunderlist, which I don't actually use just sent me the following in email:
Apache won't start after reboot
I have a VM in EC2 via the HUB, based on the LAMP appliance. Everything was running fine (except for the bug vulnerability).
Then I ran:
# install-security-updates
# /etc/init.d/apache2 restart
Apache failed.
I rebooted, still could not start Apache.
The error log says:
So I looked at some of the comments above, then I ran:
# /usr/lib/inithooks/firstboot.d/15regen-sslcert
# /etc/init.d/apache2 start
And still get the same failure.
Any hints or help would be greatly appreciated!
Check your permissions
Check the permissions of your ssl certs. They should be readable by group certssl.
If they are only readable by root, apache will complain.
Information is free, knowledge is acquired, but wisdom is earned.
Also check your sites-enabled
You didn't specifically mention which appliance you are using. Mine is a drupal7 appliance hosting a drupal6 install. In my /etc/apache2/sites-enabled/ I have config files for drupal6, drupal7 and phpmyadmin. Both drupal configs have an entry for SSLCertificateFile pointing to the aforementioned cert.pem. If you also have a default-ssl, it may be using a different cert, /etc/ssl/certs/ssl-cert-snakeoil.pem. Make sure you have a SSLCertificateFile setting in one of your available sites-enabled and that it points to the TurnKey generated cert.pem with the correct ownership and permissions.
Information is free, knowledge is acquired, but wisdom is earned.
Thanks for the comments. VM
Thanks for the comments. VM was a standard LAMP, v 13.0.
The problem seems to have been something with the old certificates. Got things working creating a new VM, restoring from backup, then deleting the Apache virtual domains and recreating them with the new (default installed) certificates. Old certs were in place, with seemingly correct permissions, but not working. Got it working just before going on vacation.
Now might be a good time to implement Forward Secrecy
With all the attention that HeartBleed has brought to security, now might be a good time to think about implementing Forward Secrecy, sometimes referred to as 'Perfect' Forward Secrecy or PFS in all TurnKey appliances that use OpenSSL. Forward Secrecy protects against the future compromise of your private keys revealing the contents of prior communications. Forward Secrecy is already implemented in OpenSSL but is usually not configured.
After applying the HeartBleed updates and generating a new set of self-signed certs, I looked into implementing PFS. I'm currently running nginx configured as a reverse-caching-proxy and setup to handle all SSL requests for my backend apache servers. The configuration allows me to handle multiple domains while having a single external IP address.
"Enabling forward secrecy can be done in two steps:
I followed the configuration advice for nginx from Qualys Security Labs (links below). I did have to modify their example slightly to get nginx to accept it (doesn't like '\'). My nginx config now looks like this:
ssl_params
ssl_prefer_server_ciphers tells the server to choose a matching cipher from the list in the order given. Specifying each cipher explicitly makes sure that the most secure, highest performance ciphers are given highest priority, but the list may need to be updated from time to time. I chose to include SSLv3 to support older clients even though it does not support Forward Secrecy.
References:
https://community.qualys.com/blogs/securitylabs/2013/06/25/ssl-labs-depl...
https://community.qualys.com/blogs/securitylabs/2013/08/05/configuring-a...
Information is free, knowledge is acquired, but wisdom is earned.
Great suggestion John
I have added it to the TKL Issue tracker. Thanks! :)
Article about the guy who inadvertently introduced the bug
I just came across an interesting article in the Sydney Morning Herald quoting public statements made by Dr Robin Seggelmann regarding his inadvertent introduction of the 'Heartbleed' bug into OpenSSL. Have a read here.
I find it particularly interesting that whilst he takes responsibility for introducing the bug, he ultimately blames lack of resources for the OpenSSL software as being the main factor... Raises the valid point that if you care about free open source software then people need to put their money and/or their time where their mouths are! :)
After all steps TKL 13 LAMP install still has 1.0.1e installed
I have a TKL13 LAMP installation that I am trying to update but the affected OpenSSL doesn't seem to be updated. As you can see below it is still running a February release of 1.0.1e (or so it seems). The server has been rebooted several times and going apt-get upgrade shows 19 upgrades available. Fortunately my server is Internal only but still concerning that what seems that it should work isn't. Unfortunate because it can't get to the testers. Thoughts?
root@lamp ~# install-security-updates
+ SEC_UPDATES=FORCE
+ /usr/lib/inithooks/firstboot.d/95secupdates
Hit http<SNIPPED>
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@lamp ~# uname -a && cat /etc/*release
Linux lamp 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux
PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"
NAME="Debian GNU/Linux"
VERSION_ID="7"
VERSION="7 (wheezy)"
ID=debian
<SNIP>
root@lamp ~# openssl version
OpenSSL 1.0.1e 11 Feb 2013
root@lamp ~#
Sorry
If I would have added the -a switch to openssl version it would have told me it was built on Apr 8th. Looks like it did update then.
Steve
Hi Steve
Glad you worked it out. The info you show in your first post does appear a little misleading and I think your confusion is not unreasonable. However if you look further up this thread then you can see a discussion that covers the upgraded OpenSSL package, that may (hopefully) make it a little clearer.
FYI a line that I find handy for finding out info about packages is:
Thanks!
Thanks for sharing!
Pages
Add new comment