Core & TKLDev v16.0 Release Candidates - now available for download!

Update 2020-05-04: We've finally published Core and TKLDev as stable releases (v16.0); along with a few others. The RC releases noted here are no longer available for download.
Please see the first instalment of updated appliances.
Update 2020-05-15: Please see the second instalment of updated appliances.
Update 2020-05-20: Please see the third instalment of updated appliances.
Update 2020-05-25: Please see the forth instalment of updated appliances.
Update 2020-06-09: Please see the fifth instalment of updated appliances.
Update 2020-06-29: Please see the sixth instalment of updated appliances.
Update 2020-07-16: Please see the seventh instalment of updated appliances.
Update 2020-07-27: Please see the eighth instalment of updated appliances.
Update 2020-08-27: Please see the ninth instalment of updated appliances.
Update 2020-08-31: Please see the tenth instalment of updated appliances.
Update 2020-10-19: Please see the eleventh instalment of updated appliances.

Update 2020-03-17: Due to reported bugs in TKLDev's firstboot startup procedure, we have released a TKLDev v16.0rc2 ISO. The text and links below have been updated to reflect his new RC.

With great excitement, it is my pleasure to announce that the highly anticipated and well overdue release candidates of the next major TurnKey Linux version. TurnKey Core, v16.0rc1 is now available for download. For avid community developers who would like to get involved, we also have a v16.0rc1 v16.0rc2 release of TKLDev.

Core and TKLDev

As per all previous release candidate (RC) releases of TurnKey major versions, we have published Core (our base appliance; that all others are based on) to give the community an opportunity to test out the next major version of TurnKey. Hopefully that will give you all a chance to give the base OS a trial and let us know what you think. We especially welcome constructive criticism.

As per the last few RC releases, we've also published TKLDev; our "build tool in a box". The idea behind that is that users and developers who are super keen to get going with v16.0 can do some testing and perhaps even push things ahead for their favourite appliance(s). If you're not familiar with TKLDev, there is a tutorial on How to build appliance ISOs with TKLDev. It was actually written for v14.x, but was updated for v15.x and should still apply to v16.0 too. There are also a number of TKLDev related docs available in the docs directory on GitHub. It's worth noting that some of these docs are out of date but we hope to get them up to date ASAP. If you hit any issues, please feel free to ask. If you'd like to help out with doc improvements and/or updates, please do not hesitate to open a pull request!

Regarding development of v16.0 appliances, please note that a number of appliances are already done, or close to it. However, at the time of writing, none of the code has yet been merged into the master branch of the relevant GitHub repositories. I hope to start doing that within the coming week or so. So it's important to ask and/or double check on GitHub for unmerged pull requests relating to v16.0. Regardless, it's highly desirable if you share what you're working on and how you are progressing. Hopefully that will reduce the likelihood of people treading on each other's toes so to speak and/or wasted efforts on anyone's behalf.

Download the RCs and help us test them

Core (64bit / amd64 build)314MB ISO  ( changeloghash filemanifest )

TKLDev (64bit / amd64 build)373MB ISO  ( changeloghash filemanifest )

Quick overview of significant changes

I'll do a full outline of the changes when we do the final stable release of v16.0, but here's a quick overview of major changes:

  • Rebase on Debian 10/Buster.
  • Updated ISO install splash screen.
  • Major refactoring of Di-Live, including porting to Python3, update of d-i source components (pulled direct from Debian d-i source) and now leverages Debian's default live-boot & live-tools (no longer using our own fork of casper and busybox).
  • Most other TurnKey custom software ported to Python3, including Confconsole and Inithooks (and their dependencies). Wherever possible, TurnKey custom software now depends on default Debian packages (rather than our own forks; e.g. dialog & python3-dialog).
  • Webmin - Update to latest v1.941, with latest stable version of the 'Authentic' theme.
  • Key rotation and individual keys for specific purposes (i.e. new key for images, and one new key for each TurnKey apt repository; 'buster', 'buster-security' & 'buster-testing').
  • Some quite significant changes to TKLDev (complete rewrite of the bootstrap creation process; no longer including pool, chanko and friends OOTB).

  • Considering how long it's taken, on face value the list doesn't appear that significant. However, the work behind the scenes to date has been huge. Much of our effort has gone to bringing TurnKey back into line with Debian and the move to Python3. We have also done some significant work on removing (or at least reducing) internal blockages and bottlenecks, so hopefully future work can be a bit quicker and smoother (we'll need to wait and see how successful we've been there...).

    There is still plenty to be done including quite a bit that I'd love to see included in v16.0. On the other hand, I don't really want to hold up the release of v16.0 stable appliances any longer than I have to, so at least some of that will almost certainly be delayed until later in the lifetime of v16.x. Regardless, we'll try to get as much in as we can whilst still pushing v16.0 out the door ASAP.

    Please help test and give us your feedback

    As with previous "point-oh" releases, this is a major OS transition. As such, we need plenty of testing! We've already done a fair bit of testing in house, but the more the merrier! So hopefully you get a chance to give the RCs a test drive. We welcome all feedback, especially constructive critique. Please post below, open a new thread in the forums (requires a free website user account) or bugs can be reported directly to the Issue tracker (free GitHub user account required). Look forward to hearing from you! :)


    Jeremy Davis's picture

    Thanks tons for having a look and great to hear your feedback.

    It seems like the initial setup is/was a little fragile. The fact that it didn't set itself up properly initially is a bit weird and the LAMP failure you noted (which worked later) is also a little weird. I'll certainly keep an eye out for those and if I can recreate them, I'll try to work out what is going on...

    Really look forward to hearing your thoughts about the apps you built. Thanks again for jumping in to help out!

    Jeremy Davis's picture

    Great work! Looks like you've nailed the issue to the tkldev-setup script not being able to verify the bootstrap. I've opened an issue on the tracker for that. I'll make that a priority as that will cause us issues...

    TKLDev issue

    To answer your question re the importance of the issue, I guess it doesn't really matter, as manually setting up those repos resolves the issue. However, IMO it's an issue well worth fixing ASAP for a few reasons:

    • The bootstrap isn't being verified. I guess it's fairly unlikely that someone malicious is distributing a hacked bootstrap which includes malware, but I think it's best to be sure!
    • I don't think that a buggy setup really inspires confidence in TurnKey as an appliance building framework. Obviously bugs happen in all software, but the less the better - and to hit one so quickly is far from ideal...
    • The learning curve for the uninitiated might already feel a bit steep. Hitting an issue like this so early in the process might be just enough to scare a new comer off.
    • And perhaps most importantly from our perspective, we use TKLDev ourselves to build all the appliances. When we do a build run, we do that on a clean instance of TKLDev. So requiring manual intervention to build an appliance is a deal breaker!

    MySQL/MariaDB background

    For the benefit of anyone reading this post, some background and context should probably be noted. First up, it should be explicitly noted, that as of v15.0, TurnKey provides MariaDB, rather than actual MySQL. MariaDB is a MySQL fork, which is intended as a "drop in MySQL replacement". It was created by the original developer/founder of MySQL, along with many of the MySQL dev team who jumped ship when Oracle bought out Sun (who owned MySQL). You can red more about the history on Wikipedia.

    As of 9/Stretch Debian have removed MySQL completely and provide MariaDB instead. We (in TurnKey v15.x) followed Debian's lead, who in turn are following the lead of other Linux distros such as Red Hat, and use the terms "MySQL" and "MariaDB" interchangeably (so all the old MySQL commands work, but you're actually using MariaDB).

    Just to confuse things, Ubuntu haven't followed that lead of other Linux distros and provide both MariaDB and MySQL packages as clear and distinct software options (as it seems you are aware). It's well worth keeping that in the back of your mind if applying an Ubuntu tutorial to TurnKey - as Ubuntu is based on Debian, often Ubuntu tutorials are relevant, but unlike Ubuntu, TurnKey is 100% binary compatible with Debian. It's perhaps also worth noting, that whilst MariaDB is a drop in replacement for MySQL, once you start using MariaDB, it's not guaranteed that you can go back to MySQL. That is because MariaDB contains additional features which MySQL (at least the free open source version) doesn't support. If you're interested, you can read more about MariaDB <--> MySQL compatibility. Anyway, from here on in I'll continue to use the terms "MariaDB" and "MySQL" interchangeably (and to specifically refer to MariaDB as included in TurnKey).

    MariaDB/MySQL root user with no password

    So to get back to your concern re no password. The change to use of 'unix_socket' (aka 'auth_socket') authentication for the root MySQL user account (as opposed to 'password' authentication) became the default for MariaDB v10.4.3+. FWIW my reading suggests that it is also the default for MySQL v5.7+ too.

    Whilst on face value, not having a password for the root user account may seem like a huge concern, IMO it's actually a security improvement! It's been the Debian default since the introduction of MariaDB in 9/Stretch. TurnKey followed suit and kept that default in v15.0 onwards.

    It sounds like your research has given you a bit of an understanding of it, but let me explain how it works. In Debian prior to 9/Stretch (& TurnKey prior to v15.0), a password was used to authenticate the root MySQL user via a network connection. In TurnKey as an additional security measure, the root account was locked to localhost.

    However since Debian 9/Stretch (& TurnKey v15.0); a unix socket is used instead. A unix socket is sort of like a special file. This socket is owned by the relevant Linux user (so in this case, it's owned by the root Linux user). This means that only the root (Linux user) user can access the root MySQL user account, thus limiting access to the specific user. Another feature of unix sockets is that they are only available via localhost (you can't ever authenticate via a unix socket remotely; unless via a method such as SSH where you are essentially accessing localhost via a secure remote connection). The theory is, that the security & authentication mechanism for the root Linux user login should be sufficient to stop unauthorised access to the localhost system, and therefore the root MySQL account.

    I would argue that's a legitimate assumption to make. If someone untrusted has root access to your server, then access to MySQL DBs is probably the least of your problems!

    Another factor is that many additional system functionalities of MySQL require root access. E.g. starting/stopping the MySQL service, rotating the MySQL logs, cron jobs/maintenance scripts which compress, cleanup and/or otherwise optimise your DBs, etc. The way that was resolved previously was via a special "root equivalent" system user account (named "debian-sys-maint"). However, that user had it's password stored within /etc in plain text! Whilst this has historically been somewhat common practice (using file permissions to ensure that only the root user has read access to this plain text password) in more recent times, it's been recognised as a potential security weakness and most apps are moving away from that model. AFAIK, newer MySQL releases support storage of encrypted passwords (so no more plain text), but 'unix_socket' authentication is generally considered superior.

    MySQL password set on firstboot & final words

    The MySQL password you are asked to set on TurnKey LAMP (and LAMP based appliances) firstboot is for the "adminer" MySQL user account. That is a root like user too (so you can do DB administration via Adminer) but as it's available via a web interface, it requires a password.

    I can't really speak to why Ubuntu have chosen to do things differently. As I noted above, AFAIK use of socket authentication is the default for both MariaDB v10.4.3+ & MySQL v5.7+. FWIW the MySQL v8.0 docs (under "Socket Peer-Credential Authentication") note:

    [...] auth_socket authentication is well suited to server administration user accounts for which access must be tightly restricted.

    I hope that sets your mind at ease...

    Jeremy Davis's picture

    I noted what I've done and linked to the git diffs in a comment on the tracker.

    FWIW the main issue was that tkldev-setup had the GPG key hardcoded into it and we've rotated the keys for Buster. The updated script now clones the repos first and gets the key from the Buildtasks config. So future iterations should "just work" (so long as the key ID has been updated in buildtasks).

    Unfortunately, because the script is contained within the overlay of the TKLDev build code, it won't be included until the next release. I'll probably need to do a RC2 build of TKLDev to fix that...

    In the meantime, if you wanted to, you could download it like this:

    wget $URL/$FILE -O /$FILE

    It probably won't do much if you run it as you've already manually set it up. But if you wanted to test it out, you could remove the unpacked bootstrap and test that part (it will automatically pull the latest commits for all the repos before it checks the bootstrap downloads). Obviously, you'll need to download the updated tkldev-setup script first (as per above), then remove the Buster bootstrap and re-run it like this:

    rm -r /turnkey/fab/bootstraps/buster

    FWIW, here's what I got when I just re-ran it (after just removing the unpacked Buster bootstrap as per above):

    INFO [tkldev-setup]: /turnkey/buildtasks exists, attempting update.
    Already up to date.
    INFO [tkldev-setup]: /turnkey/tklbam-profiles exists, attempting update.
    Already up to date.
    INFO [tkldev-setup]: /turnkey/fab/cdroots exists, attempting update.
    Already up to date.
    INFO [tkldev-setup]: /turnkey/fab/common exists, attempting update.
    Already up to date.
    INFO [tkldev-setup]: /turnkey/fab/products/core exists, attempting update.
    Already up to date.
    INFO [tkldev-setup]: Downloading bootstrap-buster-amd64
    File 'bootstrap-buster-amd64.tar.gz' already there; not retrieving.
    File 'bootstrap-buster-amd64.tar.gz.hash' already there; not retrieving.
    INFO [signature-verify]: Verifying GPG signature
    gpg: Signature made Sun Mar  8 10:37:48 2020 UTC
    gpg:                using RSA key F190A48B54DC56B2C7F24DCBAC5EB00493E5BC1C
    gpg: Good signature from "TurnKey GNU/Linux Buster Images (GPG signing key for TurnKey Linux Buster Images) " [unknown]
    gpg: WARNING: This key is not certified with a trusted signature!
    gpg:          There is no indication that the signature belongs to the owner.
    Primary key fingerprint: A8B2 EF42 8781 9B03 D351  6CCA 7623 1C20 425E 9772
         Subkey fingerprint: F190 A48B 54DC 56B2 C7F2  4DCB AC5E B004 93E5 BC1C
    INFO [signature-verify]: GPG verification success.
    INFO [signature-verify]: Verifying checksum.
    INFO [signature-verify]: Checksum verification success.
    INFO [tkldev-setup]: Unpacking bootstrap-buster-amd64
    INFO [tkldev-setup]: tkldev-setup complete.
    Jeremy Davis's picture

    Thanks to your feedback Ian, we've just published an updated v16.0rc2 TKLDev. Hopefully it should all "just work" now! :)

    Jeremy Davis's picture

    Great work, thanks so much Ian! :)

    I'm currently working my way through some bugs that I've discovered in other build formats (e.g. OVA and LXC). As soon as I have all the other builds working, I'll start releasing v16.0 stable appliances.

    Thanks again, your help is invaluable.

    Jeremy Davis's picture

    I only just noticed this post (for some reason I only get notifications for forum posts?!).

    Anyway, I'm really hoping to have v16.0 LAMP and a few others released this week. However, we've had some internal infrastructure issues and I've been diverted to deal with that as a matter of urgency. I'm still hoping to get them out this week, but I can't promise anything...


    Add new comment