Automatic security updates

Overview

TurnKey automatically installs the latest security updates over the network:

  1. The first time you boot a new appliance deployment (you can choose to skip this)
  2. Every night, around 4 AM.

Usually automatically updating software is considered to be a risky practice since updates may occasionally break existing functionality (e.g., changes to file formats, software interfaces, or expected behavior).

Debian mitigate this risk by carefully backporting security fixes so that security updates change as little as possible, minimizing the likelyhood that things will break.

In practice we've found it is very rare for a security update to break something, so we believe it is beneficial to  configure software appliances to auto-update security fixes by default. Advanced users can always disable this mechanism and apply security fixes manually if they want.

Installing security updates on demand

In a root shell, run the following command:

turnkey-install-security-updates

If that doesn't work you may be running an older version of TurnKey. Try this instead:

/usr/sbin/cron-apt

Caution: This isn't 100% full-proof. Make sure we can reach you.

Unfortunately, we can't fix everything automatically so it's still very important that we be able to contact you when necessary. Make sure you're subscribed to TurnKey's low-traffic announcements newsletter. (Requires registration for a free website user account, "Security and News Announcements" subcriptipon checkbox should be ticked. If already registered, check the "subscriptions" tab of your user account info).

Otherwise you may not know that a problem requires your attention until it's too late. Sure, thanks to automatic security updates we usually don't need to bother you regarding security issues, but there are occasional exceptions...

  1. Not everything can be updated automatically: automatic security updates only work for supported software that is maintained using the package management system. Not all software is installed through the package management system. Not all software installed through the package management is supported.  See the limitations section below for details.
     
  2. Some bugs can break automatic updates: even though security updates change as little as possible and are exceptionally well tested, mistakes can still happen. Usually these can be caught and fixed with another automatic update, but manual intervention is still required for bugs that break the auto-updates mechanism or one of its dependencies (e.g., Ubuntu broke cron).

How it works

Users who wish to tweak the auto-update mechanism may find it helpful to understand how it is set up.

1) A cron job is configured to run cron-apt daily.

# cat /etc/cron.d/cron-apt
#
# Regular cron jobs for the cron-apt package
#
# Every night at 4 o'clock.
0 4     * * *   root    test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt

2) cron-apt is configured to only update from the security sources list.

# cat security.sources.list 
deb http://archive.turnkeylinux.org/debian buster-security main

deb http://security.debian.org/ buster/updates main
deb http://security.debian.org/ buster/updates contrib
# deb http://security.debian.org/ buster/updates non-free

3) cron-apt is configured to install the updates automatically:

$ cat /etc/cron-apt/action.d/5-install
autoclean -y
dist-upgrade -y -o APT::Get::Show-Upgraded=true \
                   -o Dir::Etc::sourcelist=/etc/apt/sources.list.d/security.sources.list \
                   -o Dir::Etc::sourceparts=nonexistent \
                   -o DPkg::Options::=--force-confdef \
                   -o DPkg::Options::=--force-confold

4) cron-apt logs to /var/log/cron-apt/log

Issue resolution options

In some very rare situations[1], trying to install a security update silently, a conflict or problem may occur, such as an unfullable dependency. In that scenario, with default config, TurnKey automatic updates auto-uninstalled MariaDB. Not ideal, although at least you know there's a problem... The upside of the default behaviour is that you'll most likely be fully aware of the issue. The downside is that it might be because your server crashed in some way.

[1] As far as I'm aware, the only time this has occurred in TurnKey servers was Nov 2018 (v15.0 IIRC).

As of v16.1, TurnKey Confconsole includes the option to resolve the issue via an alternate strategy. The alternate strategy will never remove any packages; if a security update can't install, then it will leave the (unpatched and potentially insecure) app as is. The upside is that everything should continue to run on your server. The downside is that may include applications with unpatched security issues.

Whichever strategy you use, you will need to be aware of the inherent risk trade-offs. And either way, be sure to make sure we can reach you so we can keep you in the loop around any manual steps required to resolve the issue.

Limitations

TurnKey Linux is based on Debian (generally/ideally latest version matches Debian stable). The Debian Security Team provides backported security fixes for all packages in Debian as required, which TurnKey systems are configured to automatically install (as per this page).

However, Debian's security coverage does not apply to packages that do not originate from Debian.

  • Trusted third party repositories: ideally the Debian package repositories would cover 100% of our software needs. Unfortunately in practice there's a lot of good software out there that Debian does not package. In these cases, TurnKey will install software directly from trusted third party repositories.

    Note that any packages that does not originate from Debian/TurnKey repositories is documented on the appliance product page, and also in the product's source code.
     
  • TurnKey Linux custom packages: TurnKey contains a few custom packages which are updated directly by the Core developers from the project's cryptographically signed package repository.
     
  • Software installed from source code: unfortunately, many of the most popular open source web applications (e.g., Joomla) are not packaged by Debian. This means that they have to be installed and maintained by hand directly from upstream source code and no automatic security updates can be provided through the package management system.

    Fortunately, most web applications run with reduced privileges and are developed in high-level programming languages that are less susceptible to many of the most serious low-level security vulnerabilities. Also in the appliance model, each application is confined to its own virtual machine. This limits the potential damage somewhat but vigilance is still recommended, especially for high-risk usage scenarios.

    When a TurnKey appliance includes software installed from upstream source code, this is usually the first thing documented on the appliance page.

You can use the "apt policy" command to determine a package's origin. Note that you should generally run "apt update" prior to ensure that your local package index is up to date wit the repository servers.

# apt policy openssh-server
openssh-server:
  Installed: 1:7.4p1-10+deb9u6
  Candidate: 1:7.4p1-10+deb9u6
  Version table:
 *** 1:7.4p1-10+deb9u6 500
        500 http://security.debian.org stretch/updates/main amd64 Packages
        500 http://http.debian.net/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

So in the case of openssh-server, we have the most recent version installed and are receiving updates automatically from the security.debian.org repository.

Comments

L. Arnold's picture

As I am normally needing to setup tests on static ip's, it would be helpful if the TKLBAM and Update setup could be specified after the IP address is entered.  The one time I tried to do it before seemed to cause problems, but once built it is not obvious always where to get the auto updates going (maybe I haven't looked that hard).  TKL BAM is easy to start and I would rather not backup too soon.

Would it be possible to move these two steps after "network configuration" is complete?

Jeremy Davis's picture

So at worst your appliance will be without the latest security updates until 4am. But as long as you have a DHCP running and accessable to your appliance, the updates on first boot should work regardless of whether you change the IP later or not. All the appliance needs is access to the internet and a valid IP address to do the updates.

Not quite sure what you're asking for in relation to TKLBAM, but it shouldn't matter what order its done in and if you'd rather do it after setting a static IP then you can do that from Webmin pretty easily.

If that still isn't meeting your needs I have some other ideas of how you could execute those scripts so it would run again but I'd rather wait until I've got access to a TKL appliance so I know it will work.

Probably best to post in the forum with a clear outline of what you are trying to acheive and why.

L. Arnold's picture

I tried to run the script "above" but wants to run at a "randomized" time around 4:00 am.

Instead I ran:

"apt-get update" and it seemed to do something similar to a security update

Not trying to accomplish anything other than having a complete install before 4 AM.

Not a big deal.  My DMZ does not have dhcp running on it so I can't easily toggle.  Seems my firewall will only facilitate dhcp (itself) on the inside network  -  short of time to set up a server just for a few dhcp addresses on the DMZ.

mishav's picture

So they have automatic updates for other services that are not initially automatically installed like BIND?


Jeremy Davis's picture

Any package installed from package management that has a secuirty update provided in the Debian security repo will get a security update every day!

Jeremy Davis's picture

TurnKey Linux has it pre configured, but it shouldn't be too hard to config it from scratch. If you google "cron-apt" then you should plenty of info relevant to your distro.

Alon Swartz's picture

Thanks for reporting the issue, this was a serious blunder upstream.

The regression was fixed, went through QA and new packages published within 9 hours of the bug being reported on LP. During that time the buggy packages were removed from the archive to prevent further breakage.

If you were affected, you should restart the cron service and update to the latest packages.

Jeremy Davis's picture

Although I have read (on the bug report) that a new (fixed) update has now been released. Anybody who got the dodgey update will however need to at least restart cron as cron will not start the auto update (to collect the fixed update) until that is done. Probably doing a manual update would be a good idea anyway.

[update] Alon beat me. Got sidetracked with my browser window open and thats what happens :)

Jeremy Davis's picture

Thanks for the great suggestion!

I have added it as a feature request on the issue tracker and added an entry in the docs as well.

Jeremy Davis's picture

And for going the extra distance to actually trackdown the bug report you link to. I have lodged a TKL bug report and hopefully Liraz (or perhaps Alon) will fix it asap.

Liraz Siri's picture

I upgraded that server last week so I guess I'm going to have to be the one to fix it. I think when I tested it right after the upgrade it was serving up a cached page from our web accelerator. That must have been why I didn't see the error...

Liraz Siri's picture

Argh! Thanks for reporting this. We recently upgraded that server. Seemed to work fine when I tested it. Will fix this ASAP.

Liraz Siri's picture

Alright I upgraded the wiki software to be compatible with PHP 5.3. All better now.

Jeremy Davis's picture

But obviously make sure you have a (tested working) backup first.

I would imagine that it should manually upgrade smoothly (although I have no knowledge of Moodle). In my experience upgrading minor versions (eg 2.3.1 to 2.3.2) is generally trouble free. Often even upgrading from versions like 2.3.x to 2.4.x can often go smoothly (although obviously depends on the project and how they define the difference between different version numbers). Major version upgrades (such as 2.x.x to 3.x.x) almost always involve some tweaking, although many projects provide migration and/or upgrade scripts.

Be great if you could document your progress and post in the forums (even if it's a failure). Others could probably learn from your experience and perhaps even help if you have issues.

Jeremy Davis's picture

cron-apt will only update security fixes. Wereas apt-get dist-upgrade will update everything.

Generally cron-apt is almost foolproof, apt-get upgrade is the next least likely to break things as it only applies the incremental updates to the existing packages. dist-upgrade will update to the latest version of packages avaiable via the repos so has the greatest chance of breaking something.

As a general rule they all should work fiine, but I wouldn't be inclined to run dist-upgrade unless you have time to test (and potentially fix) everything that updates, especially on a mission critical system.

Unless it's broken, it usually doesn't need fixing IMO.

Jeremy Davis's picture

It is generally easier to fix issues from a running system than from one that won't boot. A bit late for that though by the sounds...

I would suggest that you try fixing grub2 from a live cd (any Linux one should be fine - if you're unsure of how, google should turn up plenty of results - instructions for Ubuntu and/or Debian should work ok).

While you're at it, I'd run an fsck on the /boot partition and on the LVM just in case there is some corruption there.

For future reference, my number one advice is that if things that aren't broken they don't need fixing! So don't go there unless you are prepared to fix them (after you've broken them). My suggestion two is that if broken (or may be broken), things are generally easier fixed while still running (don't reboot unless you're sure everything is as it should be).

PS if you need more help probably better to start a new thread in the support section rather than hijacking this one.

Jeremy Davis's picture

And probably from a 'best practice perspective. Also aptitude isn't installed by default (but you could easily install it with apt-get if you wanted). IIRC Debian (and Ubuntu too?) suggest that apt-get is the preferred commandline package management app (rather than aptitude).

However AFAIK both apt-get and aptitude leverage apt (which in turn leverages dpkg, dselect and others), so at a system level they are basically doing similar stuff. On the flip side, I think that they use their own (i.e. different) databases to keep track of manually installed packages, thus where concerns arise...

Also I administer a couple of Proxmox VE (Debian based) servers and as production hypervisors I feel much safer using aptitude safe-upgrade when running system upgrades (as opposed to apt-get upgrade).

Having said that, I have also used apt-get on them at times to install individual software (just force of habit, not neccessarily intentionally planned that way) and have yet to experience any issues because of mixing aptitude and apt-get (although who knows... perhaps it will come back to bite me one day...)

Out of interest your question prompted me to do a bit of research and I found that there are a couple of factors at play... Firstly it seems that there have been bugs in both packages in the past that have caused many of the concerns to arise. Secondly they handle dependancies a little differently so you may get different results depending on which you use.. Plus more...

Here are some interesting quotes from my reasearch (although I suggest that if you prefer to use aptitude then you do your own research and make your own decision/conclusion):

There is a lot of BS and outdated info going arround on apt vs aptitude there are actually a couple of seperate issues.

One is tracking of unused packages, back in sarge it used to be that only aptitude did this and due to a bug in that tracking aptitude got in a mess if you used any other package manager (this bug is the source of most of the outdated advice not to mix apt-get and aptitude). In etch that bug was fixed but still only aptitude would track unused packages. In lenny and later tracking of unused packages was moved into libapt so other frontends also support it. There are differences in how they use it though, aptitude tries to remove unused packages as soon as they become unused while apt-get waits for the user to specifically request unused packages are removed.

The other is dependency resoloution. Aptitude uses it's own dependency resoloution system. Apt's system is simple, predictable and gives up quickly when it can't find a soloution. Aptitude's is more complex, less predictable and sometimes gives soloutions that are nothing like what the user asked for but it is more likely to find a soloution.

The only real difference is Aptitude.

  • If you use it interactively install something, then remove that package in something else and then go back to Aptitude, it will think you want to reinstall it. You just have to clear selections when it loads (easy enough through the menu).

  • It will also run an autoremove so old dependencies are cleaned up. This can be dangerous if you accidentally remove something that is a dependant of a metapackage and you remove it and all its deps. This isn't an issue if you know what you're doing.

I often switch back and forth between using `apt-get` and `aptitude`. I put `apt-get` in scripts or use it from the command-line, but I usually use `aptitude` for browsing, searching, and installing packages by hand. So, is it OK to mix the different apt front-ends such as `apt-get` and `aptitude`? The answer is, "sort of". It's harmless for installing packages, but if you remove a package that was installed by one of the other front-ends then they can get confused and you might end up with packages remaining installed that you don't need or having packages that you manually installed automatically removed for you. Each front-end keeps its own separate database of which files you manually specified for installation and the package dependencies that were installed automatically to satisfy your manual request. So later, if you decide to remove a package, the front-end uses its database to decide if it should remove the dependency packages or not. The problem is they each have their own database that isn't shared.

One solution is to use 'apt-mark-sync'. I generally use `aptitude` most of the time, so I treat `aptitude` as the master. I run this command to keep the other front-ends in sync:

apt-mark-sync aptitude all
Casey Iiams-Hauser's picture

Hi there, 

TK12 seems to be removing tomcat 6 every night on my servers. This might be for a very good reason, but it's not really what I want to happen. I'd really love some n00b help to disable the update. 

I'm not the best with full understanding the cron jobs, and I can see where some of the script locations are, but please point me to the one file where I can comment out the cron job to keep it from running:

EG: nano /etc/cron

something like that (yes, I also use nano, because vi is hard)

 

Best,

Casey

EDIT: but I'll leave this up because the tomcat thing is super wacky 

/etc/cron.d/cron-apt

is where the file lives. 

Casey Iiams-Hauser, MIA                 
Informatics Implementation Specialist

International Training & Education Center for Health (I-TEC

Jeremy Davis's picture

I now use vim and it is a very powerful text editor, but for the average joe that isn't working in a text editor lots there is nothing wrong with nano (I used it for years before I was 'forced' to use vim!)

Anyway, TBH I'm not sure why your server is removing Tomcat. My guess is that it is applying a security update to a Tomcat dependency (that Tomcat is incompatible with so is automatically being removed as your system considers that the better of 2 evils). So whilst on face value stopping it from doing security updates resolves your issue, there is obviously a good reason why it is doing that (i.e. like I say there is security issue). So probably a better longer term resolution is troubleshoot the actual issue and see if you can do a workaround that doesn't involve allowing your server to be vulnerable to attack.

Another option might be to just update to TKL v13.0. TKLBAM should be able to migrate your data from v12.x to v13.0 (you may need to do some minor tweaks so test first).

Obviously if your server is only available locally (i.e. via LAN/intranet and not accessible over the internet) then you can safely ignore my advice.

Keyturns's picture

FYI - The link "TurnKey's low-traffic announcements newsletter" (https://www.turnkeylinux.org/announcements) gives a 500 error (link is in section Caution: This isn't 100% full-proof. Make sure we can reach you)

Jeremy Davis's picture

I just spotted this post! Argh! Right you are! I've opened an issue and will look into it ASAP. Thank for reporting and apologies on ridiculously slow response...!

Dan's picture

Under the title "There are no dumb questions"... could someone please let me know that if I run the security updates manually as stated above, will this update my root certificates?

I have received notification from UPS that we are using a soon to be expired certificate to connect to their system.  Our production shipping system is running on a Turnkey LAMP box that daily updates have been disabled on.   

We disabled the security updates after one of them caused the system to fail. 

Thank you!

Jeremy Davis's picture

Firstly, the only certificate updates that you'd likely get via the auto security updates would be the ca-certificates package and considering that hasn't been updated anytime recently, I doubt there is an update available (security related or otherwise). FWIW the "ca" stands for "Certificate Authority" - so in the context of TurnKey those certs are the details of the trusted CAs that tools such as wget and curl use to check against when you connect to https URLs. Note that they only apply to whether the CA is trusted by default or not, the actual certificates themselves will be provided by the https end point.

So without knowing more details about the way that the certificate is being used to connect to this service, it's hard to guess how you might go about getting a new and/or updated certificate; or even if it's something you can do from your end?! Personally, I suggest that you contact them and enquire about the process and check if there is anything you need to do; and if so, what.

If they follow up and note that you need to take further steps and you are unclear how to do those steps on TurnKey, please open a fresh thread in the forums and I'll do my best to help out.