Jacob Papenfuss's picture

The OTRS developers are kind enough to have an update scoreboard available when you first log into OTRS to see what patching needs to be done. This image was supposedly updated August 21, 2012...


OTRS 3.1.12 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.1.11 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.17 is available! Please update now. (Release Note - Level: Patch)
OTRS 2.4.15 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.1.10 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.16 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.1.9 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.1.8 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.14 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.1.7 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.1.6 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.1.5 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.1.4 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.1.3 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.12 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.1.2 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.1.1 is available! Please update now. (Release Note - Level: Major)
OTRS 3.0.11 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.10 is available! Please update now. (Release Note - Level: Security)
OTRS 3.0.9 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.8 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.7 is available! Please update now. (Release Note - Level: Security)
OTRS 3.0.6 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.5 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.4 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.3 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.2 is available! Please update now. (Release Note - Level: Patch)
OTRS 3.0.1 is available! Please update now. (Release Note - Level: Major)
OTRS 2.4.12 is available! Please update now. (Release Note - Level: Patch)
OTRS 2.4.10 is available! Please update now. (Release Note - Level: Security)

For some reason I'm not buying it, and sure not using it. In fact, I've gone and deleted the other handful of disk images I grabbed, because holy crap... Sure, maybe apt-get update was run, but this thing's 21 months out of date and counting for the actual software package featured.

I appreciate the effort that goes into these images (otherwise I'd not be trying to avoid doing it myself), but this one's a bit much, isn't it? 

Jeremy Davis's picture

You seem to lack understanding of how Debian works. Debain is built for stability, not to have the 'latest and greatest version'. Due to extensive (and I mean extensive) testing (which takes months and months, sometimes years) by the time Debain stable is released it is not uncommon for software versions to be over a year (sometimes nearly 2 or more years) 'out of date'. But they don't ignore security issues. Debian package maintainers work tirelessly to backport security patches to the specific versions in the Debian repos. So yes, by doing a simple version check, on face value it would appear that the 2.x version of OTRS (as included in TKL OTRS appliance) has many security bugs - but as is often the case, the face value is in fact incorrect.

Sometimes a new version may have new features or improved functionality and an old version (in the Debian repos) may have a few (non-security) bugs, but the version from the Debian repos will be rock solid stable and as I said above will have all the security patches backported and applied (TKL auto checks for them on first boot and again every night and auto installs them as soon as they become available). So security is a non issue with Debian packages (despite what upstream may report on a simple version check).

As far as other TKL appliances go, it does sometimes get a little tricky. When Debian packages are so out of date that they lack basic required functionality or sometimes the software isn't already packaged by Debian at all. In these cases TKL devs install from upstream. Appliances with upstream software are a little more problematic as they will require manual updating to reduce security risks but many users successfully stick with the 'out of date' versions and apply patches and/or tweaks themselves to mitigate security issues and keep regular (automated) backups using TKLBAM. If any issues arise, they simply restore a recent backup to a clean appliance instance. However this situation doesn't apply to the OTRS appliance (if you check on the indivdual appliance page you can see whether the software is from Debian repos or upstream). If you have any troubles working out where specific software comes from, I am happy to help you uncover how that knowledge can be aquired.

So for future reference could I suggest that it's probably better to be a little less aggressive and avoid spouting until you have all the facts. Otherwise you risk coming across as a bit of a tool. Generally if something doesn't make sense to me, or looks not quite right, I think it is safer to assume that I am missing something and that there is some reason behind it (that I have failed to grasp), rather than assume that those involved (users and developers) are idiots (or whatever)...

So bottom line is, if TKL doesn't work for you, that's fine, don't use it. Remeber the appliances are a gift. If you aren't interested in the gift that's your perogative\. If you have some feedback (even if it's critical) that's fine too, in fact it's encouraged. But please present it respectfully because my perception of the attitude you present above can get people like me (that devote a lot of time and effort to this project) offside instantly. And it's just not neccessary... With kinder more measured words, you could have raised the same point, made the same criticisms and got a somewhat similar answer; all without getting me on the defensive and making me think that perhaps you are a bit of an arrogant uninformed ass...

Jacob Papenfuss's picture


You're certainly right - I did come across as an arrogant ass (I'll address uninformed briefly), and I do apologize for that. My goal was definitely to box some ears, so to speak, but I failed to actually deliver the message I intended to. A bad day, all around, and it was vented in the wrong direction.

I'm quite familiar with Debian's nature, and just didn't bother to look closely enough at the image or its packages to see that it was a Debian-maintained package. It was the first thing I saw after firstboot, where the firstboot update process failed because I don't have a DHCP server active on the network the image was on, and the network configuration menu is displayed after the package update. I'm assuming that if you have DHCP available, a lease is grabbed so the update works - otherwise, that goes from fairly bad to something worse.

I'll shift my complaint to something that is definitely TKL maintained: the security vulnerabilities in Webmin 1.590, which is installed and enabled by default. If the user views the Webmin login page, they'll see a prompt to update provided by upstream directly - but if they don't do this, the vulnerabilities go unpatched. There doesn't appear to be anything in the TKL repo for security updates to webmin, and the packages available are all timestamped before the disclosure of the vulnerability at first glance - there might be something else mitigating the issue, but I didn't bother investigating further since my threshold for potential issues was hit before I got that far. 

I'll also admit that my alarm is not in any way diminished by the implication that reimaging and restoring from backup is an acceptable response to a successful attack. Restoring the image won't accomplish anything until the vulnerability is patched, since you'll still be running vulnerable software, and restoring from backup assumes that no other tampering or modifications happened, and that they aren't present in the backups that would be restored. 

As for the comment about gifts, it's doubly fitting to mention the gift given to the Trojans. That aside, when the first thing I see are two different security vulnerability notices, one of which is a bundle of 4 remotely exploitable holes (and 2 that don't require authentication), I don't think that it's out of line to cut my losses. I don't know why I decided to drive by and flip everyone the bird on my way out, exactly, but here we are.

As a postscript, I decided to see just where one can find the version of Webmin installed by default in each image, which led me to the changelog for Core, which includes this statement: "Disabled inline webmin upgrades (managed by APT)." - This isn't true in the images I've tried, thankfully. It is also the only place I could easily find a version number, aside from delving into package manifests. And the only other TKL image I still have is SiTracker, which is running a very nearly unmodified 3.65, with two security related releases appearing to be unapplied. 

Jeremy Davis's picture

I sincerely appreciate your acknowledgement and apology. You are obviously a man of substance and I can appreciate that in the heat of the moment things can be said which may not come across all that well (ahh the intracicies of online communication...)

So despite your initial (partial) misinterpretation of the OTRS situation and my initial somewhat defensive reaction to your OP, you do raise some valid points.

As you correctly note, the images get their IP etc via DHCP (at least initially) and as you also correctly note the initial security update occurs prior to the opportunity to set a static IP. So I can see how you would think this is a bad thing. Personally I think this is mitigated by a few factors.

  1. When the appliance fails to get an IP (and/or cannot connect to the internet) the init update script clearly notes (at least it has in my experience) that the security update failed and suggests that you run 'cron-apt' to rerun the security updates once you have manually configured networking. Obviously while the appliance has no networking, there is no security risk.
  2. As I said (in my post above), the security updates are auto configured to run every 24 hours (randomised time around 4am). So within less that 24 hours the update scripts will run again. I guess that if your appliance doesn't run 24 hrs then you may not get these auto updates - so it's not foolproof.
  3. In fairness to TKL, it should be noted that no other publicly available Debian server appliance that I am aware of auto runs security updates - so normally the user/admin would need to do this manually anyway.

You also raise a valid point with Webmin. I am unclear whether the TKL devs have backported the security patches for Webmin, but my suspicion is that they haven't. However my additional suspicion is that in this instance they have opted to prioritise user-friendliness/functonality over security (whether that is a fair and reasonable concession is obviously debatable). You may have noticed that one of the prided and unique features of TKL appliances is TKLBAM (the custom 'smart' backup and migration utility). Many users have stated that this is their main reason for choosing TKL appliances. As part of the functionality of TKLBAM (to make it more user friendly) the core TKL devs provide a Webmin module to access the TKLBAM features (rather than making it commandline only, or having to create and maintain their own webUI for it). This means that if Webmin is updated, there is a risk that the TKLBAM Webmin module could break thus making TKLBAM unreliable and/or unusable for those relying on Webmin to acess it.

Additionally, the Webmin Security Alerts documents only 2 current vulnerabilities in the Webmin version included with TKL. From my reading, the first requires a somewhat complex setup involving viewing of a malicious website immediately prior to logging in to Webmin to compromise the system. I.e. more that a straightforward external exploit that can be applied by an unauthenticated hacker alone. The other requires the atacker must have control of the root domain (where the TKL appliance and it's Webmin are available) and then also requires viewing of a malicious page. Both of these relate to the File Manager module (which you correctly state can be updated from within Webmin). So whilst not a completely ideal situation, it is something of a measured risk. Personally I don't use Webmin much so generally disable it (and reenable it if/when I want to use it). I generally advise users to do the same (for any service that they don't use - eg Webmin, Webshell, phpMyAdmin, etc) and/or enable the (IPTables) firewall (and configure it with the relevant ports closed - by default it is disabled, but configured with all the relevant ports opened). Not only does that approach minimise current issues, but also as yet undiscovered ones.

Out of interest, the "Disabled inline webmin upgrades (managed by APT)." you mention relates to the core Webmin app (rather than the individual tar.gz modules themselves). This was a response to a previous situation where if the user updated Webmin to the latest version (from within Webmin) an update loop situation could occur. If apt-get upgrade was run, then the original Webmin would be reinstalled from the repos and the 'Update Webmin" alert would again occur (from within Webmin itself). Also as I mentioned above, this also introduces the risk that the TKLBAM module would become unfunctional or unstable.

Your point regarding the restoration of a backup and the implications are also valid - although only to a pont IMO (obviously the vulnerabiliy still remains). Your point that the restored backup may contain 'tampering or modifications' that affect security in some cases may possible, although IMO generally this is (at least somewhat) mitigated by the fact that TKLBAM is not an image - it contains only the user data. The core app files are vanilla TKL and untouched (assuming that you restore to a clean instance as I would always recommend - especially if your site has been hacked). It is still possible that infected files are contained in the user data that is backed up, but i have not yet come across anyone who has reported that experience when restoring/migrating to a clean/fresh appliance (not to say it's not possible - just speaking from my experience of manning these formums for 3-4 yrs). Obviously though that may not be good enough for you and fair enough too. It is also worthy of note IMO, that this strategy has been advocated as a 'quick and dirty trick' by a number of users who use the Joomla appliance (Joomla seems to be reknowned for security issues and frequent updates - hence I assume why no one is brave enough to package it in Debain) - not as general reponse to any and all security concerns. I also note that TKLPatch could be used to update the appliance and/or TKLBAM allows the usage of pre and post installation hooks which could be utilised to update the upstream software if so desired. 

So all-in-all you raise a number of valid security points (regardless of my mitigations). But as with anything in life there are always tradeoffs (we can't have it all). And unfortunately due to the limited resources of the core devs, some potential security risks have been trumped by other concerns/developments. Whether that is an acceptable situation or not is (as always) the decision of the end user. There is the option to modify it to suit your standards, or as you have demonstrated - leave it alone altogether.

To put the situation into a little context, I have come to know the core devs quite well over my years here and I know that their dreams for TKL are nowhere near fulfilled (they still see it as very much a work in progress) and that if they had unlimited resources that they would dearly love to do much, much more. They have never stated that TKL is the holy grail of hardened high-security server appliances - they have stated they they believe that they are near-enough for Linux newbs to get started and use as-is (this point paraphrased by me), or a good starting point for professionals with higher expectations and/or needs.

As with any other opensource project, anyone with enthusiasm, knowledge and willingness to contribute, additional resources (by way of manpower especially) are always warmly welcomed. There are at least a couple of community contributed patches to update and/or mitigate issues in current specific TKL appliances (and a number of others to provide additional software which TKL doesn't currently include). I know that one of the dev's current priorities is to streamline, upgrade and open up their build infrastructure to allow the community more involvement in developing and addressing some of the current TKL shortcomings. Unfortunately though they have not yet released that, so we still have to work with what we have, namely TKLPatch - to allow for the updating and adjusting of the current TKL images. Currently TKLPatch is only configured to work with ISOs or (with reservation) installed filesystems, but I have also documented how they can be applied to OVZ templates.

I would again like to acknowledge you readiness to apologise for the not so constructive tone of your initial post and the more measured and thoughtful approach to the constructive critisms you have raised in your second. Ideally, I would love it if rather than dump the images, you were interested in engaging and assisting to address them, (by way of developing TKLPatches) but I understand that that may not be a practical option for you and your current personal/professional constraints. Reagrdless, on behalf of TKL (although I am only a volunteer) I thank you for taking the time to detail some of the flaws you see in TKL.

Also I have posted a link to this thread on a recent blog post by the devs speaking about and requesting feedback.

Thanks again and good luck in your future endevours! :)

Jacob Papenfuss's picture

I'm in no way a(n experienced) developer, and my security knowledge is definitely firmly entrenched in the 'putting out other people's fires' category, rather than anything on a professional level, and much more a 'pro-sumer' or higher-end hobbiest than anything, so I'm generally hesitant to try to contribute code-wise. I know wrong when I see it, but there's a pretty big gulf ahead between that and knowing what's right. 

When I look at the project (as someone who found it, oh, a week ago or so), I imagine many newbies deploying it on VMWare Player (which doesn't like the bundled config files as it doesn't recognize Debian as a supported OS - Go figure) or on VPSes and the like. They're the ones I generally have in mind when looking at things like this. For my part, I was poking at it to cycle through different helpdesk/issue trackers and... Well, bravely ran away based on first impressions.

So... On the general idea of an acceptable window of vulnerability:

I imagine most TKL deployments being amateurs in VMs (and behind NAT), or perhaps professionals in a lab, or at least behind some sort of locked down firewall, and with that in mind generally agree with the idea that a 24ish hour window is reasonable given the low visibility level of (my perceived) average TKL install. Where it becomes more problematic is if it's deployed as-is to a production environment - but ideally you're at least moving a prepared system into production... In any case, it is a small window. The bigger issue are out of date packages maintained by TKL.

Webmin 1.590 had 4 different CVE identifiers issued against it, although they are pretty similar. CERT lists 3, and rolled them up together at http://www.kb.cert.org/vuls/id/788478

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-2981 - Arbitrary perl execution for authenticated users (service monitor scripts)
http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-2982 - Arbitrary remote program execution by authenticated users (show.cgi)
http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-2983 - View contents of files on filesystem, no authentication required (edit_html.cgi)
http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-4893 - CSRF/session hijacking for different scripts that interact with system binaries or compression (This one was probably merged with another, as it's much less common than the others, and no patch exists for it specifically). 

One requires a bit of setup for CSRF against the admin, 2 require authentication against Webmin, 1 was exploitable without authentication. 

It's definitely a package that needs to be updated, or if a backport was made, the package needs a version increment to reflect this. It doesn't help matters that half the CVEs say 1.580 and older, when it was 1.590 and older (or was it?). That said, I diffed Webmin upstream's 1.590 tarball and diffed the install versus original source, and it looks to be unpatched.


As for the entire restoring backup thoughts - let me give you a very direct example of an exploit that would survive anything you did to the filesystem, so long as you left the database alone - which is pretty much exactly what such a backup restore would do.

phpBB3 has a feature that lets you cache your template/style data in your database instead of on disk. It also has a feature that allows (nearly) arbitrary PHP to be included in these template files. I don't know what sequence of exploits allowed the attacker to arrange this configuration, but the board admin was stumped when even with a clean tarball, following the install procedure, and then restoring their database,  and running on a read-only filesystem, they were being attacked.

I found the exploit code in one of the primary template files for Subsilver. It was really braindead, yet brilliant at the same time - The entirety of the exploit was an addition to the template file that would try various PHP functions like passthru, system, etc until it found one that worked, and would just pass the contents of a HTTP GET variable. So,

http://example.com/?knockknock=yes&cmd=ls -lR /

would execute for the attacker, and if the GET variable wasn't set, it displayed the page as normal. One phpBB setting was modified to accomplish this, and since they were restoring database dumps and reinstalling from scratch, they were at a loss and were trying to find an OS vulnerability when I finally got them to let me look at their logs. If the attacker had used a particular session ID passed through a cookie or something even more subtle, it'd have been damn hard to find - The use of GET gave it away in the URLs logged. 

Sure, the problem would have been much less severe had they disabled the usual dangerous functions, ran with more strict permissions and firewalled outbound as well as inbound (attacker was rsyncing files in and out of their install directory, though, and was already restricted to files owned by the user PHP was running as). 

And that wasn't taking into account things like Wordpress templates that come with exploits built in, which the user would probably turn around and immediately reinstall if their reimaged instance didn't have it after restoring from backup.

There seems to be a great need for an audit of packages provided in these images and what's available in the repository for updates. Without knowing exactly what the TKL package maintainers have modified and in what way, that makes the job that much harder. (backports, customised installs that may mitigate attacks (or create new vulnerabilities, such as Directory Indexing being on for SiTracker's image, with the file attachment directory being a static directory under the document root, and no access restrictions via .htaccess - If you happened to find a stock SiTracker TKL install in the wild, you have free access to documents attached to issues without authentication - But I can't find any sites with both "powered by turnkey linux" and "sitracker" in their bodies to prove it)). Add to it that if you look at package versions and filenames, they don't match the software installed, but often take the TKL version number they were packaged for:

	user@sitracker ~$ aptitude show turnkey-sitracker-12.0
Package: turnkey-sitracker-12.0
State: installed
Automatically installed: no
Version: 1
Priority: optional
Section: misc
Maintainer: Alon Swartz <alon@turnkeylinux.org>
Uncompressed Size: 4096
Description: turnkey-sitracker-12.0 release

I downloaded the original 3.65 package for this and diffed it against the TKL install. TKL has a patch upstream doesn't, but it was one for a minor error message - It adds another variable to be passed to a function, and removes a newline after the PHP close tag. 

		root@sitracker ~/sit-3.65# diff /root/sit-3.65 /var/www/sitracker --suppress-common-lines  --recursive
Only in /var/www/sitracker: attachments-f9d6a1eb922dcf63cc78dbde5f580332
Only in /var/www/sitracker: config.inc.php
diff --suppress-common-lines --recursive /root/sit-3.65/incident_details.php /var/www/sitracker/incident_details.php
< $win = clean_fixed_list($_REQUEST['win'], array('','incomingview', 'jump', 'holdingview', 'sit_popup'));
> $win = clean_fixed_list($_REQUEST['win'], array('','incomingview', 'jump', 'holdingview', 'sit_popup', "incident{$id}"));
< ?>
\ No newline at end of file
> ?>

And there's no updated .deb available in TKL's repository, just the original "12.0" release. 

Jeremy Davis's picture

I appreciate the time you have taken with your response. To be honest, you seem pretty competent to me! TBH you put me to shame... :)

Again I think you raise great points, I was especially interested to read that the exploits (as noted on the Webmin site) appear to be out of date and/or incomplete.

Your example of the potential usage of the DB for cacheing (phpBB) is a great point (that I was completely unaware of) and duly noted...

You're right, some sort of auditing tool would be fantasic! A web app included on the website would be a great hing I reckon (which included an automated comparrision of the contents of the image vs what's available in the enabled repos) plus the ability to add notes manually (perhaps sort of like a wiki or comments section). Unfortunately I don't see that happening anytime soon....

Add new comment