TurnKey Linux Virtual Appliance Library

Automatic security updates

TurnKey automatically installs the latest security updates over the network:

  1. The first time you boot a new appliance deployment (you can choose to skip this)
  2. Every night, around 4 AM.

Usually automatically updating software is considered to be a risky practice since updates may occasionally break existing functionality (e.g., changes to file formats, software interfaces, or expected behavior).

Debian mitigate this risk by carefully backporting security fixes so that security updates change as little as possible, minimizing the likelyhood that things will break.

In practice we've found it is very rare for a security update to break something, so we believe it is beneficial to  configure software appliances to auto-update security fixes by default. Advanced users can always disable this mechanism and apply security fixes manually if they want.

Installing security updates on demand

In a root shell, run the following command:

install-security-updates

If that doesn't work you may be running an older version of TurnKey. Try this instead:

/usr/sbin/cron-apt

Caution: This isn't 100% full-proof. Make sure we can reach you.

Unfortunately, we can't fix everything automatically so it's still very important that we be able to contact you when necessary. Make sure you're subscribed to TurnKey's low-traffic announcements newsletter.

Otherwise you may not know that a problem requires your attention until it's too late. Sure, thanks to automatic security updates we usually don't need to bother you regarding security issues, but there are occasional exceptions...

  1. Not everything can be updated automatically: automatic security updates only work for supported software that is maintained using the package management system. Not all software is installed through the package management system. Not all software installed through the package management is supported.  See the limitations section below for details.
     
  2. Some bugs can break automatic updates: even though security updates change as little as possible and are exceptionally well tested, mistakes can still happen. Usually these can be caught and fixed with another automatic update, but manual intervention is still required for bugs that break the auto-updates mechanism or one of its dependencies (e.g., Ubuntu broke cron).

How it works

Users who wish to tweak the auto-update mechanism may find it helpful to understand how it is set up.

1) A cron job is configured to run cron-apt daily.

# cat /etc/cron.d/cron-apt
#
# Regular cron jobs for the cron-apt package
#
# Every night at 4 o'clock.
0 4     * * *   root    test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt

2) cron-apt is configured to only update from the security sources list.

# cat security.sources.list 
deb http://archive.turnkeylinux.org/debian wheezy-security main

deb http://security.debian.org/ wheezy/updates main
deb http://security.debian.org/ wheezy/updates contrib
# deb http://security.debian.org/ wheezy/updates non-free

3) cron-apt is configured to install the updates automatically:

$ cat /etc/cron-apt/action.d/5-install
autoclean -q -y
dist-upgrade -q -y -o APT::Get::Show-Upgraded=true \
                   -o Dir::Etc::sourcelist=/etc/apt/sources.list.d/security.sources.list \
                   -o Dir::Etc::sourceparts=nonexistent \
                   -o DPkg::Options::=--force-confdef \
                   -o DPkg::Options::=--force-confold

4) cron-apt logs to /var/log/cron-apt/log

Limitations

TurnKey Linux 13 is based on Debian 7.0 (Wheezy). The Debian Security Team provides backported security fixes for all packages in Debian as required, which TurnKey systems are configured to automatically install.

However, Debian's security coverage does not apply to packages that do not originate from Debian.

  • Trusted third party repositories: ideally the Debian package repositories would cover 100% of our software needs. Unfortunately in practice there's a lot of good software out there that Debian does not support. In these cases, TurnKey will install software directly from trusted third party repositories.

    Note that any packages that does not originate from Debian is documented on the product page, and also in the product's source code.
     
  • TurnKey Linux custom packages: TurnKey contains a few custom packages which are updated directly by the Core developers from the project's cryptographically signed package repository.
     
  • Software installed from source code: unfortunately, many of the most popular open source web applications (e.g., Joomla) are not packaged by Ubuntu or Debian. This means that they have to be installed and maintained by hand directly from upstream source code and no automatic security updates can be provided through the package management system.

    Fortunately, most web applications run with reduced privileges and are developed in high-level programming languages that are less susceptible to many of the most serious low-level security vulnerabilities. Also in the appliance model, each application is confined to its own virtual machine. This limits the potential damage somewhat but vigilance is still recommended, especially for high-risk usage scenarios.

    When a TurnKey appliance includes software installed from upstream source code, this is usually the first thing documented on the appliance page.

You can use the "apt-cache policy" command to determine a package's origin:

# apt-cache policy openssh-server
openssh-server:
  Installed: 1:6.0p1-4+deb7u1
  Candidate: 1:6.0p1-4+deb7u1
  Version table:
 *** 1:6.0p1-4+deb7u1 0
        500 http://security.debian.org/ wheezy/updates/main amd64 Packages
        500 http://cdn.debian.net/debian/ wheezy/main amd64 Packages
        100 /var/lib/dpkg/status

So in the case of openssh-server, we have the most recent version installed and are receiving updates automatically from the security.debian.org repository.

Comments

L. Arnold's picture

Sequencing of "Setup"

As I am normally needing to setup tests on static ip's, it would be helpful if the TKLBAM and Update setup could be specified after the IP address is entered.  The one time I tried to do it before seemed to cause problems, but once built it is not obvious always where to get the auto updates going (maybe I haven't looked that hard).  TKL BAM is easy to start and I would rather not backup too soon.

Would it be possible to move these two steps after "network configuration" is complete?

Jeremy's picture

Auto security updates run everyday at 4am anyway.

So at worst your appliance will be without the latest security updates until 4am. But as long as you have a DHCP running and accessable to your appliance, the updates on first boot should work regardless of whether you change the IP later or not. All the appliance needs is access to the internet and a valid IP address to do the updates.

Not quite sure what you're asking for in relation to TKLBAM, but it shouldn't matter what order its done in and if you'd rather do it after setting a static IP then you can do that from Webmin pretty easily.

If that still isn't meeting your needs I have some other ideas of how you could execute those scripts so it would run again but I'd rather wait until I've got access to a TKL appliance so I know it will work.

Probably best to post in the forum with a clear outline of what you are trying to acheive and why.

L. Arnold's picture

4:00 AM works

I tried to run the script "above" but wants to run at a "randomized" time around 4:00 am.

Instead I ran:

"apt-get update" and it seemed to do something similar to a security update

Not trying to accomplish anything other than having a complete install before 4 AM.

Not a big deal.  My DMZ does not have dhcp running on it so I can't easily toggle.  Seems my firewall will only facilitate dhcp (itself) on the inside network  -  short of time to set up a server just for a few dhcp addresses on the DMZ.

So they have automatic

So they have automatic updates for other services that are not initially automatically installed like BIND?


Jeremy's picture

Yes

Any package installed from package management that has a secuirty update provided in the Debian security repo will get a security update every day!

sir can i install in this in

sir can i install in this in a new system

 

Jeremy's picture

You certainly can

TurnKey Linux has it pre configured, but it shouldn't be too hard to config it from scratch. If you google "cron-apt" then you should plenty of info relevant to your distro.

warning: security update with buggy PAM breaks CRON (stopped)

Hi, a quick note to warn everybody that as of a few hours ago (31st May 2011, 23:00 GMT) ubuntu security updates where distributing a PAM update that breaks CRON and therefore any macinhe running any meaningful operation based on CRON will not perform correctly.

https://bugs.launchpad.net/ubuntu/+source/pam/+bug/790538

I'm immediately disabling security updates (low risk host; stable monitoring very very important)

Take care

SM

Alon Swartz's picture

Update: Fixed

Thanks for reporting the issue, this was a serious blunder upstream.

The regression was fixed, went through QA and new packages published within 9 hours of the bug being reported on LP. During that time the buggy packages were removed from the archive to prevent further breakage.

If you were affected, you should restart the cron service and update to the latest packages.

Jeremy's picture

Thanks for the heads up

Although I have read (on the bug report) that a new (fixed) update has now been released. Anybody who got the dodgey update will however need to at least restart cron as cron will not start the auto update (to collect the fixed update) until that is done. Probably doing a manual update would be a good idea anyway.

[update] Alon beat me. Got sidetracked with my browser window open and thats what happens :)

Proxy

Had an issue where this wasn't happening - there's a daemon running that is configured by /etc/apt/apt.conf/01turnkey which causes apt to pull it's config from there rather than an exported variable or whatever (unless you reboot I guess).  Adding:

Acquire::http::Proxy "http://your.proxy.here:port";

Should resolve the problem.

Core should allow proxy setting

Many of us use TKL behind a corporate firewall that requires a proxy to get to the outside world.

It would be very helpful if the install/config system would ask "Do you use a proxy for http/https/ftp access?", then allow that to be entered (one for all, or individually) before the attempt to fetch latest updates. This isn't hard to do post-facto:

vi /etc/apt/apt.conf.d/01turnkey

then insert:

Acquire::http::Proxy "http://your.proxy.here:port/";

However, having this requested and set by the installer might save many new to TKL and perhaps an unfamiliar distro a few headaches and lost time.

 

Jeremy's picture

Great suggestion!

Thanks for the great suggestion!

I have added it as a feature request on the issue tracker and added an entry in the docs as well.

broken link in article

The wiki link in this article is broken. You have (the classic) problem that mediawiki (where your wiki is based upon) is no longer compatible with the installed PHP version.. the mentioned link: http://wiki.turnkeylinux.org/Custom_packages the error i recieve Parse error: syntax error, unexpected T_PAAMAYIM_NEKUDOTAYIM, expecting T_NS_SEPARATOR in /var/www/dekiwiki/includes/GlobalFunctions.php on line 1163 bug report http://bugs.developer.mindtouch.com/view.php?id=6874

Jeremy's picture

Thanks for posting Paul

And for going the extra distance to actually trackdown the bug report you link to. I have lodged a TKL bug report and hopefully Liraz (or perhaps Alon) will fix it asap.

Liraz Siri's picture

My bad - didn't remember it was cached...

I upgraded that server last week so I guess I'm going to have to be the one to fix it. I think when I tested it right after the upgrade it was serving up a cached page from our web accelerator. That must have been why I didn't see the error...

Liraz Siri's picture

Argh! Thanks for reporting

Argh! Thanks for reporting this. We recently upgraded that server. Seemed to work fine when I tested it. Will fix this ASAP.

Liraz Siri's picture

fixed the wiki

Alright I upgraded the wiki software to be compatible with PHP 5.3. All better now.

Maybe an update *of* cron killed cron-apt?

Hi, I'm a fellow sysadmin looking for folks who can verify whether cron-apt is actually smart enough to daemonize itself or something so that the parent 'cron' process doesn't kill it if apt tries to update cron itself.

Last night my simple cron job (based on apt-get, not cron-apt) took down various daemons to update them... including cron itself... and of course left a half-updated system and a bunch of stopped daemons. This sounds a lot like what you describe here as "Ubuntu broke cron." Only I don't think they really broke it - I think they just updated it, and cron-apt is alas no more clever than my simple cron job in this situation.

I think I need to run my job in the background with nohup so it won't die when cron dies, and make it clever enough to email its own output independent of cron.

Moodle update in TKL 12.0 ?

I have installed Turnkey Linux Moodle 12.0 and the bundled Moodle version is 2.3.1+ but from within the appliance I can see that Moodle 2.3.2+ is available. Since the bundled moodle is compiled from upstream's sources, I don't supose that it will be updated with TKL's auto update cron-apt. The question is: should I try to update manualy from 2.3.1+ to 2.3.2+ or not? My only concern is potential security updates included with this latest update.

Jeremy's picture

You could try

But obviously make sure you have a (tested working) backup first.

I would imagine that it should manually upgrade smoothly (although I have no knowledge of Moodle). In my experience upgrading minor versions (eg 2.3.1 to 2.3.2) is generally trouble free. Often even upgrading from versions like 2.3.x to 2.4.x can often go smoothly (although obviously depends on the project and how they define the difference between different version numbers). Major version upgrades (such as 2.x.x to 3.x.x) almost always involve some tweaking, although many projects provide migration and/or upgrade scripts.

Be great if you could document your progress and post in the forums (even if it's a failure). Others could probably learn from your experience and perhaps even help if you have issues.

How about apt-get update && apt-get dist-upgrade ?

Instead of running the provided /usr/sbin/cron-apt would it be better (or not?) to run

apt-get update && apt-get dist-upgrade

when deciding to manually perform the update?

Jeremy's picture

Depends what you are trying to achieve...

cron-apt will only update security fixes. Wereas apt-get dist-upgrade will update everything.

Generally cron-apt is almost foolproof, apt-get upgrade is the next least likely to break things as it only applies the incremental updates to the existing packages. dist-upgrade will update to the latest version of packages avaiable via the repos so has the greatest chance of breaking something.

As a general rule they all should work fiine, but I wouldn't be inclined to run dist-upgrade unless you have time to test (and potentially fix) everything that updates, especially on a mission critical system.

Unless it's broken, it usually doesn't need fixing IMO.

Problems updating

Hi,

 

I'm experiencing some issues with upgrade process, I have a Virtualbox based VM with only apt-cacher-ng installed on it, so when I run sudo apt-get upgrade, it runs without any visible complain; I get 4 warnings where something with grub doesn't was found. After it I restart the VM and the system just stop in the middle of booting process, it just fall at initram? prompt because it didn't  found the main bootable partition. It is happening with TKL 12.0 so there is any knowledge on how to override this issue?

Thanks in advance and sorry for my english,

Oscar.

Jeremy's picture

As a general rule, resolve upgrade issue prior to reboot

It is generally easier to fix issues from a running system than from one that won't boot. A bit late for that though by the sounds...

I would suggest that you try fixing grub2 from a live cd (any Linux one should be fine - if you're unsure of how, google should turn up plenty of results - instructions for Ubuntu and/or Debian should work ok).

While you're at it, I'd run an fsck on the /boot partition and on the LVM just in case there is some corruption there.

For future reference, my number one advice is that if things that aren't broken they don't need fixing! So don't go there unless you are prepared to fix them (after you've broken them). My suggestion two is that if broken (or may be broken), things are generally easier fixed while still running (don't reboot unless you're sure everything is as it should be).

PS if you need more help probably better to start a new thread in the support section rather than hijacking this one.

Move the install script up

Note that by default there's an "/etc/cron-apt/action.d/3-download" script, which downloads all available upgrades (but does not install them). Having the security updates install script come afterwards, will also install the already downloaded upgrades (regardless of the source setting, because they're marked as pending). This could lead to some undesired results, especially with "--force-xxx". So simply perform the security updates before you download the upgrades, by changing "5-install" to "2-install".

aptitude

if i get it right one shouldnt use aptitude for package management, because this uses apt-get and the 2 shouldnt be mixed ideally ?

thanks

Jeremy's picture

Ideally perhaps...

And probably from a 'best practice perspective. Also aptitude isn't installed by default (but you could easily install it with apt-get if you wanted). IIRC Debian (and Ubuntu too?) suggest that apt-get is the preferred commandline package management app (rather than aptitude).

However AFAIK both apt-get and aptitude leverage apt (which in turn leverages dpkg, dselect and others), so at a system level they are basically doing similar stuff. On the flip side, I think that they use their own (i.e. different) databases to keep track of manually installed packages, thus where concerns arise...

Also I administer a couple of Proxmox VE (Debian based) servers and as production hypervisors I feel much safer using aptitude safe-upgrade when running system upgrades (as opposed to apt-get upgrade).

Having said that, I have also used apt-get on them at times to install individual software (just force of habit, not neccessarily intentionally planned that way) and have yet to experience any issues because of mixing aptitude and apt-get (although who knows... perhaps it will come back to bite me one day...)

Out of interest your question prompted me to do a bit of research and I found that there are a couple of factors at play... Firstly it seems that there have been bugs in both packages in the past that have caused many of the concerns to arise. Secondly they handle dependancies a little differently so you may get different results depending on which you use.. Plus more...

Here are some interesting quotes from my reasearch (although I suggest that if you prefer to use aptitude then you do your own research and make your own decision/conclusion):

There is a lot of BS and outdated info going arround on apt vs aptitude there are actually a couple of seperate issues.

One is tracking of unused packages, back in sarge it used to be that only aptitude did this and due to a bug in that tracking aptitude got in a mess if you used any other package manager (this bug is the source of most of the outdated advice not to mix apt-get and aptitude). In etch that bug was fixed but still only aptitude would track unused packages. In lenny and later tracking of unused packages was moved into libapt so other frontends also support it. There are differences in how they use it though, aptitude tries to remove unused packages as soon as they become unused while apt-get waits for the user to specifically request unused packages are removed.

The other is dependency resoloution. Aptitude uses it's own dependency resoloution system. Apt's system is simple, predictable and gives up quickly when it can't find a soloution. Aptitude's is more complex, less predictable and sometimes gives soloutions that are nothing like what the user asked for but it is more likely to find a soloution.

The only real difference is Aptitude.

  • If you use it interactively install something, then remove that package in something else and then go back to Aptitude, it will think you want to reinstall it. You just have to clear selections when it loads (easy enough through the menu).

  • It will also run an autoremove so old dependencies are cleaned up. This can be dangerous if you accidentally remove something that is a dependant of a metapackage and you remove it and all its deps. This isn't an issue if you know what you're doing.

I often switch back and forth between using `apt-get` and `aptitude`. I put `apt-get` in scripts or use it from the command-line, but I usually use `aptitude` for browsing, searching, and installing packages by hand. So, is it OK to mix the different apt front-ends such as `apt-get` and `aptitude`? The answer is, "sort of". It's harmless for installing packages, but if you remove a package that was installed by one of the other front-ends then they can get confused and you might end up with packages remaining installed that you don't need or having packages that you manually installed automatically removed for you. Each front-end keeps its own separate database of which files you manually specified for installation and the package dependencies that were installed automatically to satisfy your manual request. So later, if you decide to remove a package, the front-end uses its database to decide if it should remove the dependency packages or not. The problem is they each have their own database that isn't shared.

One solution is to use 'apt-mark-sync'. I generally use `aptitude` most of the time, so I treat `aptitude` as the master. I run this command to keep the other front-ends in sync:

apt-mark-sync aptitude all