You are here
Automatic security updates
Contents
Overview
TurnKey automatically installs the latest security updates over the network:
- The first time you boot a new appliance deployment (you can choose to skip this)
- Every night, around 4 AM.
Usually automatically updating software is considered to be a risky practice since updates may occasionally break existing functionality (e.g., changes to file formats, software interfaces, or expected behavior).
Debian mitigate this risk by carefully backporting security fixes so that security updates change as little as possible, minimizing the likelyhood that things will break.
In practice we've found it is very rare for a security update to break something, so we believe it is beneficial to configure software appliances to auto-update security fixes by default. Advanced users can always disable this mechanism and apply security fixes manually if they want.
Installing security updates on demand
In a root shell, run the following command:
turnkey-install-security-updates
If that doesn't work you may be running an older version of TurnKey. Try this instead:
/usr/sbin/cron-apt
Caution: This isn't 100% full-proof. Make sure we can reach you.
Unfortunately, we can't fix everything automatically so it's still very important that we be able to contact you when necessary. Make sure you're subscribed to TurnKey's low-traffic announcements newsletter. (Requires registration for a free website user account, "Security and News Announcements" subcriptipon checkbox should be ticked. If already registered, check the "subscriptions" tab of your user account info).
Otherwise you may not know that a problem requires your attention until it's too late. Sure, thanks to automatic security updates we usually don't need to bother you regarding security issues, but there are occasional exceptions...
-
Not everything can be updated automatically: automatic security updates only work for supported software that is maintained using the package management system. Not all software is installed through the package management system. Not all software installed through the package management is supported. See the limitations section below for details.
- Some bugs can break automatic updates: even though security updates change as little as possible and are exceptionally well tested, mistakes can still happen. Usually these can be caught and fixed with another automatic update, but manual intervention is still required for bugs that break the auto-updates mechanism or one of its dependencies (e.g., Ubuntu broke cron).
How it works
Users who wish to tweak the auto-update mechanism may find it helpful to understand how it is set up.
1) A cron job is configured to run cron-apt daily.
# cat /etc/cron.d/cron-apt # # Regular cron jobs for the cron-apt package # # Every night at 4 o'clock. 0 4 * * * root test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt
2) cron-apt is configured to only update from the security sources list.
# cat security.sources.list deb http://archive.turnkeylinux.org/debian buster-security main deb http://security.debian.org/ buster/updates main deb http://security.debian.org/ buster/updates contrib # deb http://security.debian.org/ buster/updates non-free
3) cron-apt is configured to install the updates automatically:
$ cat /etc/cron-apt/action.d/5-install
autoclean -y
dist-upgrade -y -o APT::Get::Show-Upgraded=true \
-o Dir::Etc::sourcelist=/etc/apt/sources.list.d/security.sources.list \
-o Dir::Etc::sourceparts=nonexistent \
-o DPkg::Options::=--force-confdef \
-o DPkg::Options::=--force-confold
4) cron-apt logs to /var/log/cron-apt/log
Issue resolution options
In some very rare situations[1], trying to install a security update silently, a conflict or problem may occur, such as an unfullable dependency. In that scenario, with default config, TurnKey automatic updates auto-uninstalled MariaDB. Not ideal, although at least you know there's a problem... The upside of the default behaviour is that you'll most likely be fully aware of the issue. The downside is that it might be because your server crashed in some way.
[1] As far as I'm aware, the only time this has occurred in TurnKey servers was Nov 2018 (v15.0 IIRC).
As of v16.1, TurnKey Confconsole includes the option to resolve the issue via an alternate strategy. The alternate strategy will never remove any packages; if a security update can't install, then it will leave the (unpatched and potentially insecure) app as is. The upside is that everything should continue to run on your server. The downside is that may include applications with unpatched security issues.
Whichever strategy you use, you will need to be aware of the inherent risk trade-offs. And either way, be sure to make sure we can reach you so we can keep you in the loop around any manual steps required to resolve the issue.
Limitations
TurnKey Linux is based on Debian (generally/ideally latest version matches Debian stable). The Debian Security Team provides backported security fixes for all packages in Debian as required, which TurnKey systems are configured to automatically install (as per this page).
However, Debian's security coverage does not apply to packages that do not originate from Debian.
-
Trusted third party repositories: ideally the Debian package repositories would cover 100% of our software needs. Unfortunately in practice there's a lot of good software out there that Debian does not package. In these cases, TurnKey will install software directly from trusted third party repositories.
Note that any packages that does not originate from Debian/TurnKey repositories is documented on the appliance product page, and also in the product's source code.
-
TurnKey Linux custom packages: TurnKey contains a few custom packages which are updated directly by the Core developers from the project's cryptographically signed package repository.
-
Software installed from source code: unfortunately, many of the most popular open source web applications (e.g., Joomla) are not packaged by Debian. This means that they have to be installed and maintained by hand directly from upstream source code and no automatic security updates can be provided through the package management system.
Fortunately, most web applications run with reduced privileges and are developed in high-level programming languages that are less susceptible to many of the most serious low-level security vulnerabilities. Also in the appliance model, each application is confined to its own virtual machine. This limits the potential damage somewhat but vigilance is still recommended, especially for high-risk usage scenarios.
When a TurnKey appliance includes software installed from upstream source code, this is usually the first thing documented on the appliance page.
You can use the "apt policy" command to determine a package's origin. Note that you should generally run "apt update" prior to ensure that your local package index is up to date wit the repository servers.
# apt policy openssh-server openssh-server: Installed: 1:7.4p1-10+deb9u6 Candidate: 1:7.4p1-10+deb9u6 Version table: *** 1:7.4p1-10+deb9u6 500 500 http://security.debian.org stretch/updates/main amd64 Packages 500 http://http.debian.net/debian stretch/main amd64 Packages 100 /var/lib/dpkg/status
So in the case of openssh-server, we have the most recent version installed and are receiving updates automatically from the security.debian.org repository.
Comments
Sequencing of "Setup"
As I am normally needing to setup tests on static ip's, it would be helpful if the TKLBAM and Update setup could be specified after the IP address is entered. The one time I tried to do it before seemed to cause problems, but once built it is not obvious always where to get the auto updates going (maybe I haven't looked that hard). TKL BAM is easy to start and I would rather not backup too soon.
Would it be possible to move these two steps after "network configuration" is complete?
Auto security updates run everyday at 4am anyway.
So at worst your appliance will be without the latest security updates until 4am. But as long as you have a DHCP running and accessable to your appliance, the updates on first boot should work regardless of whether you change the IP later or not. All the appliance needs is access to the internet and a valid IP address to do the updates.
Not quite sure what you're asking for in relation to TKLBAM, but it shouldn't matter what order its done in and if you'd rather do it after setting a static IP then you can do that from Webmin pretty easily.
If that still isn't meeting your needs I have some other ideas of how you could execute those scripts so it would run again but I'd rather wait until I've got access to a TKL appliance so I know it will work.
Probably best to post in the forum with a clear outline of what you are trying to acheive and why.
4:00 AM works
I tried to run the script "above" but wants to run at a "randomized" time around 4:00 am.
Instead I ran:
"apt-get update" and it seemed to do something similar to a security update
Not trying to accomplish anything other than having a complete install before 4 AM.
Not a big deal. My DMZ does not have dhcp running on it so I can't easily toggle. Seems my firewall will only facilitate dhcp (itself) on the inside network - short of time to set up a server just for a few dhcp addresses on the DMZ.
So they have automatic
So they have automatic updates for other services that are not initially automatically installed like BIND?
Yes
Any package installed from package management that has a secuirty update provided in the Debian security repo will get a security update every day!
You certainly can
TurnKey Linux has it pre configured, but it shouldn't be too hard to config it from scratch. If you google "cron-apt" then you should plenty of info relevant to your distro.
Update: Fixed
Thanks for reporting the issue, this was a serious blunder upstream.
The regression was fixed, went through QA and new packages published within 9 hours of the bug being reported on LP. During that time the buggy packages were removed from the archive to prevent further breakage.
If you were affected, you should restart the cron service and update to the latest packages.
Thanks for the heads up
Although I have read (on the bug report) that a new (fixed) update has now been released. Anybody who got the dodgey update will however need to at least restart cron as cron will not start the auto update (to collect the fixed update) until that is done. Probably doing a manual update would be a good idea anyway.
[update] Alon beat me. Got sidetracked with my browser window open and thats what happens :)
Great suggestion!
Thanks for the great suggestion!
I have added it as a feature request on the issue tracker and added an entry in the docs as well.
Thanks for posting Paul
And for going the extra distance to actually trackdown the bug report you link to. I have lodged a TKL bug report and hopefully Liraz (or perhaps Alon) will fix it asap.
My bad - didn't remember it was cached...
I upgraded that server last week so I guess I'm going to have to be the one to fix it. I think when I tested it right after the upgrade it was serving up a cached page from our web accelerator. That must have been why I didn't see the error...
Argh! Thanks for reporting
Argh! Thanks for reporting this. We recently upgraded that server. Seemed to work fine when I tested it. Will fix this ASAP.
fixed the wiki
Alright I upgraded the wiki software to be compatible with PHP 5.3. All better now.
You could try
But obviously make sure you have a (tested working) backup first.
I would imagine that it should manually upgrade smoothly (although I have no knowledge of Moodle). In my experience upgrading minor versions (eg 2.3.1 to 2.3.2) is generally trouble free. Often even upgrading from versions like 2.3.x to 2.4.x can often go smoothly (although obviously depends on the project and how they define the difference between different version numbers). Major version upgrades (such as 2.x.x to 3.x.x) almost always involve some tweaking, although many projects provide migration and/or upgrade scripts.
Be great if you could document your progress and post in the forums (even if it's a failure). Others could probably learn from your experience and perhaps even help if you have issues.
Depends what you are trying to achieve...
cron-apt will only update security fixes. Wereas apt-get dist-upgrade will update everything.
Generally cron-apt is almost foolproof, apt-get upgrade is the next least likely to break things as it only applies the incremental updates to the existing packages. dist-upgrade will update to the latest version of packages avaiable via the repos so has the greatest chance of breaking something.
As a general rule they all should work fiine, but I wouldn't be inclined to run dist-upgrade unless you have time to test (and potentially fix) everything that updates, especially on a mission critical system.
Unless it's broken, it usually doesn't need fixing IMO.
As a general rule, resolve upgrade issue prior to reboot
It is generally easier to fix issues from a running system than from one that won't boot. A bit late for that though by the sounds...
I would suggest that you try fixing grub2 from a live cd (any Linux one should be fine - if you're unsure of how, google should turn up plenty of results - instructions for Ubuntu and/or Debian should work ok).
While you're at it, I'd run an fsck on the /boot partition and on the LVM just in case there is some corruption there.
For future reference, my number one advice is that if things that aren't broken they don't need fixing! So don't go there unless you are prepared to fix them (after you've broken them). My suggestion two is that if broken (or may be broken), things are generally easier fixed while still running (don't reboot unless you're sure everything is as it should be).
PS if you need more help probably better to start a new thread in the support section rather than hijacking this one.
Ideally perhaps...
And probably from a 'best practice perspective. Also aptitude isn't installed by default (but you could easily install it with apt-get if you wanted). IIRC Debian (and Ubuntu too?) suggest that apt-get is the preferred commandline package management app (rather than aptitude).
However AFAIK both apt-get and aptitude leverage apt (which in turn leverages dpkg, dselect and others), so at a system level they are basically doing similar stuff. On the flip side, I think that they use their own (i.e. different) databases to keep track of manually installed packages, thus where concerns arise...
Also I administer a couple of Proxmox VE (Debian based) servers and as production hypervisors I feel much safer using aptitude safe-upgrade when running system upgrades (as opposed to apt-get upgrade).
Having said that, I have also used apt-get on them at times to install individual software (just force of habit, not neccessarily intentionally planned that way) and have yet to experience any issues because of mixing aptitude and apt-get (although who knows... perhaps it will come back to bite me one day...)
Out of interest your question prompted me to do a bit of research and I found that there are a couple of factors at play... Firstly it seems that there have been bugs in both packages in the past that have caused many of the concerns to arise. Secondly they handle dependancies a little differently so you may get different results depending on which you use.. Plus more...
Here are some interesting quotes from my reasearch (although I suggest that if you prefer to use aptitude then you do your own research and make your own decision/conclusion):
TK12 Security updates is removing tomcat from my servers...
Hi there,
TK12 seems to be removing tomcat 6 every night on my servers. This might be for a very good reason, but it's not really what I want to happen. I'd really love some n00b help to disable the update.
I'm not the best with full understanding the cron jobs, and I can see where some of the script locations are, but please point me to the one file where I can comment out the cron job to keep it from running:
EG: nano /etc/cron
something like that (yes, I also use nano, because vi is hard)
Best,
Casey
EDIT: but I'll leave this up because the tomcat thing is super wacky
is where the file lives.
Casey Iiams-Hauser, MIA
Informatics Implementation Specialist
International Training & Education Center for Health (I-TEC
Nothing wrong with nano! :)
I now use vim and it is a very powerful text editor, but for the average joe that isn't working in a text editor lots there is nothing wrong with nano (I used it for years before I was 'forced' to use vim!)
Anyway, TBH I'm not sure why your server is removing Tomcat. My guess is that it is applying a security update to a Tomcat dependency (that Tomcat is incompatible with so is automatically being removed as your system considers that the better of 2 evils). So whilst on face value stopping it from doing security updates resolves your issue, there is obviously a good reason why it is doing that (i.e. like I say there is security issue). So probably a better longer term resolution is troubleshoot the actual issue and see if you can do a workaround that doesn't involve allowing your server to be vulnerable to attack.
Another option might be to just update to TKL v13.0. TKLBAM should be able to migrate your data from v12.x to v13.0 (you may need to do some minor tweaks so test first).
Obviously if your server is only available locally (i.e. via LAN/intranet and not accessible over the internet) then you can safely ignore my advice.
Link to "TurnKey's low-traffic announcements newsletter" broken
FYI - The link "TurnKey's low-traffic announcements newsletter" (https://www.turnkeylinux.org/announcements) gives a 500 error (link is in section Caution: This isn't 100% full-proof. Make sure we can reach you)
Unfortunately I missed this post until now!
I just spotted this post! Argh! Right you are! I've opened an issue and will look into it ASAP. Thank for reporting and apologies on ridiculously slow response...!
Certificate updates apply?
Under the title "There are no dumb questions"... could someone please let me know that if I run the security updates manually as stated above, will this update my root certificates?
I have received notification from UPS that we are using a soon to be expired certificate to connect to their system. Our production shipping system is running on a Turnkey LAMP box that daily updates have been disabled on.
We disabled the security updates after one of them caused the system to fail.
Thank you!
Hi Dan, I doubt it.
Firstly, the only certificate updates that you'd likely get via the auto security updates would be the ca-certificates package and considering that hasn't been updated anytime recently, I doubt there is an update available (security related or otherwise). FWIW the "ca" stands for "Certificate Authority" - so in the context of TurnKey those certs are the details of the trusted CAs that tools such as wget and curl use to check against when you connect to https URLs. Note that they only apply to whether the CA is trusted by default or not, the actual certificates themselves will be provided by the https end point.
So without knowing more details about the way that the certificate is being used to connect to this service, it's hard to guess how you might go about getting a new and/or updated certificate; or even if it's something you can do from your end?! Personally, I suggest that you contact them and enquire about the process and check if there is anything you need to do; and if so, what.
If they follow up and note that you need to take further steps and you are unclear how to do those steps on TurnKey, please open a fresh thread in the forums and I'll do my best to help out.