Jeremy Davis's picture

I've been chatting with blind_guardian23 over on Reddit. For some reason (maybe it's too long?), Reddit refused to post my most recent response and I'm over fighting with it (plus I've already spent too much time on this and I've got tons to do...). So I've taken the path of least resistance and just copy/pasted what I'd written here.

This was intended to be posted to in that thread, in the brief post that has a link here.


Thanks for your response. It looks like this one is going to be another essay. Unfortunately being brief and concise is not my strong suit...

i saw a LAMP-image with static (turnkey-) apache once

Ah! My guess is that you are confusing TurnKey with Bitnami? I'm not sure whether it's still how they do things, but Bitnami used to build cross platform installer scripts that used static binaries (built from upstream source). Their Linux builds were just a vanilla Ubuntu with their install script run on top. So the Linux build included the same (non packaged) binary installs as their Windows installers (essentially it was Linux, but with Windows-style third party software management - i.e. none!). Ever since our inception in 2008 we've used package management where ever possible. FWIW, we actually started off deb packaging PHP apps like Drupal and Joomla ourselves, but that was a full time job in itself so we had to abandon that approach.

We have NEVER, EVER shipped Apache from any source other than from official distro binary packages (early on we were Ubuntu based and installed from Ubuntu repos; in 2012 we switched to a Debian base and have installed Apache from the Debian repos since). Obviously the trade off for that, means that for most software we include, the version available is rarely ever the latest, but it is stable and supported with backported security patches - for up to 5 years (from the Debian security team initially, then from the LTS team later in the release cycle)!

In some cases, where there is a requirement for newer dependencies for a particular piece of software (e.g. PHP 8.1) we install newer versions via other methods (e.g. for newer PHP we install from 3rd party Sury apt repo; for newer NodeJS we use `n` node manager; for newer Ruby we use `rbenv`).

We do include some custom software, much of which we (deb) package ourselves.

i use unattended-upgrades on all my app servers, so security releases are installed within 24h.

FYI, we don't currently use unattended-upgrades as we have a custom `cron-apt` set up that we've been using for about 14 years. That checks daily for (Debian & TurnKey) security updates and installs them when available (it's not totally "hands off" as you'll still need to reboot under some circumstance - e.g. kernel update).

FYI `unattended-updates` didn't have a stable release when we launched back in 2008. We've re-evaluated it a few times since and whilst it's stable now and works well, we haven't yet seen enough value to change to using it (it's much bigger, slower and has many more moving parts). Having said that, I'm leaning towards replacing `cron-apt` with `unattended-upgrades` next major release because of the flexibility it provides. E.g. it's easy to change it to install all updates (rather than just security updates, which is all our current setup supports). Enabling auto reboot is also an option (which we'd still leave off by default, but it's about giving users more options).

same goes with openssl: it should ALWAYS come from upstream and with no delays, security flaws are most critical here and if a app uses old versions of it its a nightmare.

Huh? Are you suggesting that OpenSSL should not be installed via package management? I.e. should be installed from upstream (i.e. OpenSSL devs) instead of Debian packages? That sounds like a maintenance overhead nightmare! Surely you've misspoken and you meant "openssl: it should ALWAYS come from official distro repos"?!

ofc some apps (like nextcloud) do not have packages, but maybe they have a snap, appimage, docker-images or something similiar: thats fine too. If the user can update the app via web thats fine too as long it does not break possible modifications they might have made (another red flag) and user get notified of outstanding updates.

Alternate "package" mechanisms such as Snaps, Flatpaks, AppImages, Docker, etc are double edged swords. For active well maintained projects, they can be a really useful thing and make updates much easier. OTOH for poorly maintained projects they are a great way to have a ton of unmaintained and potentially vulnerable software on your system (more so the dependencies). I know that there is a degree of sandboxing which somewhat mitigates this, but a larger potential attack vector is still a larger potential attack vector (as you're likely aware, hacks are often done by daisy-chaining exploits/vulnerabilities). In my experience, they also bloat really bad over time. I recall a docker image that I used for a while that was providing a small (

Having said that, if a snap was the best way to install a piece of software, we'd use that (currently we have none). We don't use docker builds for software due to technical limitations of our (pretty dated) build infrastructure (FYI it's publicly available, rather than any philosophical or ideological reasons. Supporting software distributed via Docker is something that I hope to resolve in a future update of our build infrastructure, but no ETA. I'm still not a big fan myself, but it's not about me!

As you note, lots of software has "in app" update mechanism which is fine. The downside of that is it's near impossible to lock down a server that allows web UI updates (without changing permissions before and after). Despite that, we do support that were possible because our users have spoken. And there is an argument that being able to easily update > updating less often and running older, potentially insecure software that is more locked down. As you're no doubt aware, security isn't a binary on/off, its usually a trade-off against ease of use and/or access. Your base OS may be protected by a more locked down web app, but a vulnerable, insecure web app (even locked down) is often more easily defaced or abused by a malicious actor.

As I noted in my previous post, where there are CLI upgrade options, we provide wrapper scripts to ensure permissions remain correct. Assuming that any customizations you've made are compatible with upstream's changes and the OS itself, then upgrades should "just work".

so IF a app-provider did everything right: you are able to maintain your server yourself and it saves you time in the beginning.

That's exactly our aim!

We aim to provide software appliances that are ready to use OOTB for newbs, whilst also being a good "start point" for those with more knowledge, experience and/or bigger plans. FYI our customers/users are primarily self hosting Linux newbs (often small/micro business owners, community/non-profit groups/clubs/businesses), old school unix guys (often retired) and IT consultants (that use our images to service their customers). We also get heavily used by educational institutions as a learning tool.

At the very worst, it should be no harder to maintain our appliances than it would be if you started with vanilla Debian and installed the particular app on top using upstream recommendations (with limitations as noted earlier).

Do we always achieve that? Unfortunately not. :(

We've made lots of mistakes over the ~14 years we've been producing software appliances. And no doubt we'll make plenty more. We're also somewhat limited by upstream software providers. When we encounter software that recommends sub-optimal configuration (perhaps it works, but could be better) we may do things slightly differently (which may break reliance on upstream upgrade docs). Where ever possible we document that and/or provide tools to make it easier. Worst case scenario, we offer free support via our forums (or paid support via email). I'm not always expert in any specific piece of software, but I'm intimately familiar with Debian generally and TurnKey specifically. My Linux knowledge tends to be broad rather than deep and I love to share what I know. And if I don't know, then there's a good chance, I know who/where an answer might be found.

if not: \[...\]

Unfortunately, we have had appliances like that. But once we realize there is a better path, we migrate to it whenever possible. We try to ensure that our users (regardless of whether they are paying customers or not) don't end up stranded. If need be, I'll hand hold a user through migration to the newer/better install method and/or step through it myself to create documentation. Unfortunately, if upstream dies or goes proprietary source, etc, then there isn't much we can do about that.

i personally maintain severall ansible roles for apps and i can tell you first hand: its a lot of work, you need to support different distributions like Debian, Ubuntu, CentOS (which is now split up in different flavors), Redhat, ... which all have their specific needs. i try to re-use existing packages and other roles but still there is constant maintenance needed for new versions of the app (config changes etc.). Thats the reason there are only a few companies even offering such a product and they often aim big hosting companies which might need special customizations for their internal procedures.

Welcome to my world! :)

Although in fairness, at least I only need to work with a single (Debian stable) OS base. Having to consider alternate OS sounds like a whole new dimension of pain that I'm thankfully yet to experience! :)

Having said that, try maintaining ~100 software appliances in multiple different build formats (e.g. ISO, OVA, LXC, etc), with a very small team (currently ~2 FTE)... When you add on top of that, supporting all the customers and users we've accumulated over ~14 years in business plus infrastructure maintenance and occasional rants like this - it's a full time job and then some!

Unfortunately that means that we don't release updated apps as often as we'd like and sometimes unexpected events make that scenario even worse (e.g. we had major website issues earlier in the year which caused lots of pain - on the upside it gave me a chance for a bit of dogfooding and it went awesome). It's a problem that we don't really have a clear solution for yet, but we're hoping to focus on some automation to at least ease some of the overhead.

Despite often being older than we'd like, Debian sec updates are installed on firstboot and for most apps with an easy upgrade path, our appliances should at the very least be less work than a clean install from scratch on a fresh Debian install. FYI, as I hinted above, we are planning on focusing on some infrastructure upgrades to allow more automation (and support for docker perhaps?). We've accumulated a fair amount of technical debt over the last ~14 years and whilst we've been working through a fair bit of that over the last couple of years, we still have a way to go.

FYI, we do have customizations, but the intention is always to either improve security, improve ease of use and/or maintenance and to add to the available options/tools. Rather than create completely new processes or just to "do things our own way". Whilst it's almost always a contested concept, we do our best to implement current "best practice". When an oversight or opportunity for improvement is pointed out to us, we address it or at least share our rationale for not (which we're happy to debate with users and capitulate at times if the push back is good enough, even when it's not our preference).

if you have insight how turnkey handle this things, i would be more than happy to hear them :-)

Hopefully the above gives you some insight into our approach. Just in case it's not clear, I'm not for a second suggesting that TurnKey gets it right every time (in fact I can confirm that we make mistakes all the time). But we always seek to do our best and constantly seek feedback, especially constructive, specific, actionable user criticism. We're happy to acknowledge that TurnKey may not always be the best tool for you and/or for the job you have at hand. But we always aim to be generally useful and at worst, no harder to maintain that a server you set up yourself.

So we're far from perfect and we're not for everyone, but we're still around after ~14 years, so we must be doing something right, at least according to some people! :)

If you have any specific software appliance(s) that you'd like more explicit info on, please do ask. Hopefully I've got an answer for you, but if not I'll do my best to find it for you. Almost always, I'll be able to direct you to the specific code (on GitHub) if you really want to dig in.

Forum: 
Luke's picture

Hello! This is interesting information! Thank you for this post!
I don't understand why Reddit refused to post it, but that happens a lot there!

Add new comment