v16.0 Stable Release #1 - 10 x ISOs, Hub & Proxmox/LXC builds

Update 2020-05-15: Please see the second instalment of updated appliances.
Update 2020-05-20: Please see the third instalment of updated appliances.
Update 2020-05-25: Please see the forth instalment of updated appliances.
Update 2020-06-09: Please see the fifth instalment of updated appliances.
Update 2020-06-29: Please see the sixth instalment of updated appliances.
Update 2020-07-16: Please see the seventh instalment of updated appliances.
Update 2020-07-27: Please see the eighth instalment of updated appliances.
Update 2020-08-27: Please see the ninth instalment of updated appliances.
Update 2020-08-31: Please see the tenth instalment of updated appliances.
Update 2020-10-19: Please see the eleventh instalment of updated appliances.

With great excitement, and a sense of relief; it is my immense pleasure to announce the highly anticipated and well overdue first batch of v16.0 stable TurnKey Linux appliances. These may well be our best appliances yet?! Although, we'll allow you to be the judge of that! :)

They are all published to our mirror network and are also available for download direct from their relevant appliance pages (links provided below) or launched directly from the Hub - see more info below.

turnkey 16.0 banner

Changes of significance

Most of the significant changes in v16.x have occurred under the hood. As noted in the release candidate announcement the main changes have generally been at the software level, although much of the work will likely not be immediately obvious to an end user. In some instances, we have moved from our own forks of software (e.g. Casper for live ISO functionality) to Debian defaults ('live-boot' & 'live-tools'). In other instances, our custom software has been ported to a newer version of the language. Most of our custom software written in Python (e.g. Confconsole, Inithooks, DI-live and assorted libraries) has been ported from Python2 to Python3. This will make our custom software easier to maintain, easier to bugfix and hopefully easier for user to contribute back to (if they wish). It may also make it more attractive to other developers who may be interested in using it for their own custom purposes...

The updated appliances are now based on Debian 10/Buster and upstream software has been refreshed (where relevant). There have also been a myriad of bug fixes applied and feature requests implemented. You can view the full changes that apply to all v16.x appliances (both these and the upcoming batches) via the Core v16.0 changelog. Many of the appliances also include the changes noted in the LAMP v16.0 changelog. Each appliance has it's own changelog noting explicit changes relevant to that particular appliance (common changes are only noted in the Core changelog - otherwise each appliance notes changes relevant to it). Note that each appliances latest changelog should be available via a link on the relevant appliance page.

Launching from the Hub

Please note that for now, the Hub will still launch v15.x appliances by default (because there are only 10 v16.0 app so far). At some point, once we get a few more loaded, we'll make v16.x the default (and eventually remove v15.x altogether). In the meantime, it's really easy to launch these 10 apps (and the upcoming ones as they get added) in their v16.0 incarnation. You can click this link to launch a new v16.x server. Otherwise, look for the "16.x" text link in the top left corner (just below "logout"). It should look like this:
turnkey hub - select v16.x

Proxmox/LXC builds

We have also published these first appliances as LXC/Proxmox builds too. In case you've been living under a rock and aren't aware of these builds, they're primarily targeted at Proxmox VE but should be equally relevant to; and run fine within vanilla LXC. They can also run on Ubuntu's LXD. Unfortunately, I don't currently have access to the latest release of Proxmox; and we're yet to update our LXC appliance, but they work fine on the previous (Stretch based) Proxmox release. If you have a newer version, pelase let us know how they go. :)

The v16.0 #1 release appliances

In alphabetical order, the first 10 apps off the block are:

core appliance icon
django appliance icon
gitea appliance icon
joomla3 appliance icon
lamp appliance icon
lapp appliance icon
nextcloud appliance icon
otrs appliance icon
tkldev appliance icon
wordpress appliance icon

Where are the other apps and/or other builds?

To avoid further delays of already well behind schedule release, we decided to focus on releasing a small batch of appliances in a limited number of builds. We hope to be rolling out the rest of the library in regular batches as well as adding the other builds as we go. The VM builds should be along quite soon. The Xen and Openstack builds may initially only be released for the Core appliance due to lack of local testing equipment.

If you are able to help out with testing of any of the builds and/or have a specific appliance you'd like to see prioritised, please let us know and we'll try to get it out ASAP (no promises, but where possible, we're happy to adjust priorities).

Share your feedback

Regardless of whether you have a poor or a great experience, we'd love to hear how it goes for you. If you have general feedback, want to offer to help testing or request a specific appliance be prioritised, please post below, or open a new thread in the forums (new threads require being logged into a user account; they're free so feel free to sign up).

If you think that you've found a bug or would like to request a new feature, please feel free to skip straight over to our issue tracker (requires free GitHub user account).

I hope you like them and if not, please tell us why. Looking forward to your feedback.


Jeremy Davis's picture

Yes we fixed them so that they should now launch as unprivileged containers. A few may still not work properly unprivileged due to specific software limitations (e.g. the Domain Controller), but most should be fine.

But are you saying that you are now having issues running our containers as privileged? They should still run fine like that?! If I understand correctly, please provide more info so I can look into it.

Jeremy Davis's picture

FWIW I can confirm the issue using our v16.0 LAMP template in a privileged container on my local Proxmox server (Proxmox v5.x - based on Debian 9/Stretch).

I'd be really interested to know what Proxmox version you are running? If you are unsure, you can see it in webUI (towards the top right, to the left of the Promox logo) or by running 'pveversion'. I'd be particularly interested if it's still an issue in Proxmox v6.x.

In the meantime I've opened an issue on our tracker. I've written up details there, but I'll reiterate them here for convenience.

Firstly, apologies that I missed that. TBH, I probably should have double checked privileged containers. As it was, I only tested unprivileged ones as I mistakenly assumed that if unprivileged ones worked, then privileged containers would "just work" too. Obviously not...

So digging a little deeper, the issue appears to be a combination of LXC on the host and/or SystemD (and perhaps AppAmour too?!) on the guest. Although it primarily appears to be considered a LXC bug (at least by Debian).

Note that there is a Debian LXC bug - closed by patches which were applied to LXC to resolve this behaviour. I just checked the Debian 10/Buster LXC package to confirm that the version from the repos is fixed. As it was fixed in Debian some time ago (and is included in Buster repos OOTB) I assume that newer (i.e. v6.x / Debian 10/Buster based) Proxmox versions would not have this bug either?! They provide their own LXC package, but I assume it would likely either be running a newer version; or they would have applied the patches themselves?

Note that there is a (still open) Proxmox bug applying to Proxmox v5.x, so my guess is that this likely only applies to the older Proxmox version?! As I noted on the issue I posted though, that may not be the case?!

If anyone running Proxmox v6.x can test and confirm either way, I'd especially love to hear.

Jeremy Davis's picture

Thanks for sharing your experience. As both yourself and doxford (below) note, it looks like this bug still exists in Proxmox. :(

Also, I noted the workaround noted on the Proxmox forums/bug in the issue description but I didn't explicitly note it here. So here it is:

a reliable workaround appears to be enabling "Nesting" for the privileged container via Container -> Options -> Features -> Nesting (source: Proxmox forum thread). Note that there are security implications to this workaround (e.g. exposing the hosts /proc & /sys as read/write) so where possible, running a unprivileged container is preferable.

Unfortunately, I don't have more time to dig in deeper on this privileged container on Proxmox issue right now. Once I have published more of the library (and I've upgraded my local Proxmox server), I'll revisit this issue and see what we might be able to do on our end to improve things and/or push Proxmox to see what they can do.

If you'd like to dig in further yourself, there are some threads on the Proxmox forums that discuss ideas on alternate workarounds.

Thanks too for your report re MySQL/MariaDB password. I've opened an issue for that now too.

I'm glad to hear that you find the confconsole useful! :)

Jeremy Davis's picture

As you and peerx (bove) note, this issue does indeed still seem to affect PVE v6.x :(

As I just posted, when I get a chance, I'll dig in a bit deeper and see what else might be possible. In the meantime I also posted a workaround above, or probably best to just use unprivileged containers unless you need privileged ones.

OnePressTech's picture

Phew. Long hard road eh Jed! Thanks for all your hardwork. It is much appreciated. Quick question...any ETA on the availability of the non-Hub AMIs in the AWS marketplace?


Tim (Managing Director - OnePressTech)

Jeremy Davis's picture

Yeah it's been a monumental slog, but we're on the home stretch now. And I feel fairly confident that some of the work that has been done (particularly behind-the-scenes infrastructure stuff) will actually make things a bit smoother into the future. We'll see I guess...

Anyway to answer your question, I'm awaiting news from AWS Marketplace team whether we can do it programatically or not. If we can get that up and running within the next week or 2, we'll update them via that (and all future updates will happen much quicker than they have in the past, I would guess probably within a day of Hub updates). If that takes more than a week or 2, I'll do them manually (like we always have done historically)... FWIW the manual updates take me a bit of time to prepare on our end, then can take up to a week (occasionally more) for AWS to process (they process them manually on their end too). So fingers crossed we'll be able to (semi-)automate them. It'll make it quicker and easier for everyone...

IanH's picture

Great work on getting this out BUT what happened to the menu allowing IP address change, shut down, etc?

All I see on LAMP is "Quit" option which drops me to the command line :-(  

That is typical of the CORE install.

OK, it was my first install of the LAMP 16.0 ISO on ESXi so I'll try again to check it out but there are few options I could have done wrong!

IanH's picture


And why the Vietnamese COVID spam?

Somehow they don't have many cases but having been to Saigon in January and seen a "wet market" I am not surprised these are considered the source. Negligible hygiene but maybe they all acquire immunity by repeated exposure. Hoping I acquired some!

Jeremy Davis's picture

I didn't even bother reading it (I'm an English speaking monoglot) but I have no idea... Regardless, it's gone! :)

IanH's picture

Further checking.

LAMP15 boots to a screen with "Advanced Menu" available. SIMPLE

LAMP16 boots to a screen with just "Quit" option. That requires logging in and then the screen shows the option of an "Advanced Menu". However at subsequent boots, "Quit" option, then logging in just leaves you at the command line. There is mention of "For Advanced commandline config run: confconsole".Running that gives the screen with "Advanced Menu"!! Version 15 was so much easier. Why have a menu that just offers the option to Quit?

And NextCloud_16 is the same.

So the functionality is there but rather hidden.


Jeremy Davis's picture

It looks like I missed this post while I was writing mine...

Hopefully your questions are answered in that? As I noted, whilst it may seem a little "hidden" to you (the way that you generally use TurnKey), it should actually be "more obvious" to other users on other platforms (e.g. those that only access via SSH).

As I think I may have noted, the reason for only launching it on the first log in, is because I anticipate that it's most valuable to users on their first log in (or at least early within the lifetime of their server).

I spent a lot of time thinking, discussing and pondering how to best do this, trying to see it through the eyes of lots of different users in different usage scenarios. So I don't want to rush into changing things again. However, in the longer term, I'm not particularly attached to how it is now (just don't want to rush into changing things again until I've got some more feedback). So I am certainly open to further conversation on how it "should" work. So please feel free to share more of your thoughts.

Jeremy Davis's picture

[Oops, I posted this in the wrong place... This is a reply to this post.]

On first log in (even if via SSH), you should be met with the "complete" Confconsole. After that (as noted in the MOTD; message of the day) for future usage, you can run it from the commandline via:


It is also possible to make it run every login if you wish (but we figured the first time, with a MOTD reminder, is probably enough for most users). I still need to update the docs, but it's the "autostart " option within the Confconsole config file (/etc/confconsole/confconsole.conf). It can be "true" or "false" (the implications of whcih should be obvious). The other option is "once" which means it will run once at next log in.

TBH, I probably should have made specific note about this change, so let me do it here for now. I might move this up into the blog post, or perhaps in a new post, but for now, I'll just leave it here...

The primary reason that we changed it is that systemd (the Linux "init" system aka daemon manager) is getting really good at starting and running services asynchronously. That's great for improving boot speed, etc. But it also means that on firstboot, inithooks and confconsole were fighting for control of the terminal. That was a big issue on firstboot, but might also be an issue for later boots too (inithooks actually runs on everyboot). After spending a ton of time trying to get that working nicely, I did work out a way that we could make it work. But by then, I had also been trying to think about the issue laterally; identifying the problem we were trying to resolve and what new issues our solution may have been creating?!

So whilst I resolved the issue on how to make them start in the desired order, I also questioned why we were running Confconsole as a service anyway?! IMO there is no fundamental reason why it should be, and leveraging inithooks to launch it (when it was completed) seemed sensible. But then I realised that there was probably value in not allowing system config as root without log in. Plus, after spending so much time and energy on improving it (under the hood) it was also potentially of value to ensure that users are aware of Confconsole (at least see it once - on first login). Users who only use headless TurnKey deployment (e.g. only access via SSH, Webshell & Webmin), many users may not have even noticed that confconsole exists!

It's probably not unreasonable to allow Confconsole to run (in "complete" mode) following the firstboot questions. But to support that and allow it to also appear on first login, then that may get a bit tiresome for those users with direct terminal access. FWIW, I did also investigate leveraging PAM to allow access to the "Advanced" menu of Confconsole within Confconsole itself (i.e. log in via Confconsole rather than being dumped to the console), but I wasn't really happy with the Python PAM options.

So whilst it sounds like it comes as a bit of a rude shock to you, I hope overall, you'll agree that the new way of doing things is ok. I'd love to hear further feedback and thoughts, both form you and others...

Carlos Vercelino's picture

Hi Jeremy,

IanH has already mentioned in this post: In order to softly shutdown an appliance, now you have to loggin in, and run the "confconsole" command or the "shutdown" command.

Instead of having only the "Quit" option in the initial TurnKey Configuration Console, should it be possible to have both the "Quit" and the "Shutdown" options?

Does it make sense?

Jeremy Davis's picture

I actually missed that bit of Ian's post. Thanks for highlighting it for me!

The suggestion of a "Shutdown" option makes a ton of sense! Thanks for explicitly suggesting it!

I might even try to add this sooner rather than later, but regardless, I've opened an issue so it doesn't get forgotten.

deutrino's picture

Thank you for all the hard work!!

Would dearly love to see a Mattermost build... maybe easy since it's a Go blob like Gitea? Also a straight PostgreSQL build <3

I'm hoping to finally get some time to look into the builds I want to make now that 16.x is shipping - got a lot more going on in my business life that might converge into relearning TKLdev and hopefully creating some recipes :)

Jeremy Davis's picture

I have just finished publishing the next batch and that includes the PostgreSQL app! ;).

The 3rd batch are already in the pipeline, so it's unlikely that Mattermost will make it into that one, but we'll see what we can do re pushing that up the priority list.

Re TKLDev, we'd love to add some new apps, so any dev you want to share would be warmly welcomed. Also there have been a few TKLDev changes in v16.0, although hopefully nothing that will upset you too much! :) I still need to update the docs, but they should mostly be ok. As per always, please ask if you have questions.

Sergio_L's picture

Maybe I'm doing something wrong, but I've tried on several Turnkey LXC installations, and a couple more on bare metal and VMs just to be sure, but I'm unable to install/run any v16 container in the Turnkey LXC host, in all cases I get:

"FATAL [lxc-turnkey]: turnkey version 16.0-buster is not recognized:
lxc-create: lxccontainer.c: create_run_template: 1297 container creation template for nextcloud failed
lxc-create: tools/lxc_create.c: main: 318 Error creating container nextcloud"

If I manually specify the proper version 15 (f.e: -v 15.2-stretch) it works flawlessly, I have tried many others just to be sure it is not a nextcloud problem at all.

Am I missing something here?

Jeremy Davis's picture

Sorry to be cheeky... :)

On a more serious note, the v15.x LXC appliance only provides support for v15.x (and theoretically earlier) appliances. So it not working for v16.x guests is somewhat expected.

I have made a bit of a start on the v16.0 LXC appliance but not ETA on when it will be done. I will aim to preempt v17.x as much as I can, but it's hard to know what might be required for a future major release.

In theory, once we've updated and released the v16.0 LXC appliance you could manually download the relevant updated files if you wish and it would likely work. Although it is possible that you'd need to update to the new v16.0 release to proerly run v16.x appliances.

[EDIT] actually, after posting, I just re-read your post. It seems that the mechanism that we included in v15.x to ensure that LXC always download the latest guest appliance has had some unintended consequence. Thanks for reporting, I'll aim to keep that in mind when I do the v16.0 update.

Perhaps we should make it default to the most recent release of the same major version as the LXC host? I.e. so your LXC host would default to v15.2 Nextcloud & v15.5 Drupal8 (and would only try v16.x if you explicitly asked it to).

Or perhaps there should be a switch to specify major version?

I'd be really interested in your thoughts!


Add new comment