Core & TKLDev v16.0 Release Candidates - now available for download!

Thanks tons for having a look and great to hear your feedback.

It seems like the initial setup is/was a little fragile. The fact that it didn't set itself up properly initially is a bit weird and the LAMP failure you noted (which worked later) is also a little weird. I'll certainly keep an eye out for those and if I can recreate them, I'll try to work out what is going on...

Really look forward to hearing your thoughts about the apps you built. Thanks again for jumping in to help out!

Great work! Looks like you've nailed the issue to the tkldev-setup script not being able to verify the bootstrap. I've opened an issue on the tracker for that. I'll make that a priority as that will cause us issues...

TKLDev issue

To answer your question re the importance of the issue, I guess it doesn't really matter, as manually setting up those repos resolves the issue. However, IMO it's an issue well worth fixing ASAP for a few reasons:

• The bootstrap isn't being verified. I guess it's fairly unlikely that someone malicious is distributing a hacked bootstrap which includes malware, but I think it's best to be sure!
• I don't think that a buggy setup really inspires confidence in TurnKey as an appliance building framework. Obviously bugs happen in all software, but the less the better - and to hit one so quickly is far from ideal...
• The learning curve for the uninitiated might already feel a bit steep. Hitting an issue like this so early in the process might be just enough to scare a new comer off.
• And perhaps most importantly from our perspective, we use TKLDev ourselves to build all the appliances. When we do a build run, we do that on a clean instance of TKLDev. So requiring manual intervention to build an appliance is a deal breaker!

MySQL/MariaDB background

For the benefit of anyone reading this post, some background and context should probably be noted. First up, it should be explicitly noted, that as of v15.0, TurnKey provides MariaDB, rather than actual MySQL. MariaDB is a MySQL fork, which is intended as a "drop in MySQL replacement". It was created by the original developer/founder of MySQL, along with many of the MySQL dev team who jumped ship when Oracle bought out Sun (who owned MySQL). You can red more about the history on Wikipedia.

As of 9/Stretch Debian have removed MySQL completely and provide MariaDB instead. We (in TurnKey v15.x) followed Debian's lead, who in turn are following the lead of other Linux distros such as Red Hat, and use the terms "MySQL" and "MariaDB" interchangeably (so all the old MySQL commands work, but you're actually using MariaDB).

Just to confuse things, Ubuntu haven't followed that lead of other Linux distros and provide both MariaDB and MySQL packages as clear and distinct software options (as it seems you are aware). It's well worth keeping that in the back of your mind if applying an Ubuntu tutorial to TurnKey - as Ubuntu is based on Debian, often Ubuntu tutorials are relevant, but unlike Ubuntu, TurnKey is 100% binary compatible with Debian. It's perhaps also worth noting, that whilst MariaDB is a drop in replacement for MySQL, once you start using MariaDB, it's not guaranteed that you can go back to MySQL. That is because MariaDB contains additional features which MySQL (at least the free open source version) doesn't support. If you're interested, you can read more about MariaDB <--> MySQL compatibility. Anyway, from here on in I'll continue to use the terms "MariaDB" and "MySQL" interchangeably (and to specifically refer to MariaDB as included in TurnKey).

MariaDB/MySQL root user with no password

So to get back to your concern re no password. The change to use of 'unix_socket' (aka 'auth_socket') authentication for the root MySQL user account (as opposed to 'password' authentication) became the default for MariaDB v10.4.3+. FWIW my reading suggests that it is also the default for MySQL v5.7+ too.

Whilst on face value, not having a password for the root user account may seem like a huge concern, IMO it's actually a security improvement! It's been the Debian default since the introduction of MariaDB in 9/Stretch. TurnKey followed suit and kept that default in v15.0 onwards.

It sounds like your research has given you a bit of an understanding of it, but let me explain how it works. In Debian prior to 9/Stretch (& TurnKey prior to v15.0), a password was used to authenticate the root MySQL user via a network connection. In TurnKey as an additional security measure, the root account was locked to localhost.

However since Debian 9/Stretch (& TurnKey v15.0); a unix socket is used instead. A unix socket is sort of like a special file. This socket is owned by the relevant Linux user (so in this case, it's owned by the root Linux user). This means that only the root (Linux user) user can access the root MySQL user account, thus limiting access to the specific user. Another feature of unix sockets is that they are only available via localhost (you can't ever authenticate via a unix socket remotely; unless via a method such as SSH where you are essentially accessing localhost via a secure remote connection). The theory is, that the security & authentication mechanism for the root Linux user login should be sufficient to stop unauthorised access to the localhost system, and therefore the root MySQL account.

I would argue that's a legitimate assumption to make. If someone untrusted has root access to your server, then access to MySQL DBs is probably the least of your problems!

Another factor is that many additional system functionalities of MySQL require root access. E.g. starting/stopping the MySQL service, rotating the MySQL logs, cron jobs/maintenance scripts which compress, cleanup and/or otherwise optimise your DBs, etc. The way that was resolved previously was via a special "root equivalent" system user account (named "debian-sys-maint"). However, that user had it's password stored within /etc in plain text! Whilst this has historically been somewhat common practice (using file permissions to ensure that only the root user has read access to this plain text password) in more recent times, it's been recognised as a potential security weakness and most apps are moving away from that model. AFAIK, newer MySQL releases support storage of encrypted passwords (so no more plain text), but 'unix_socket' authentication is generally considered superior.

MySQL password set on firstboot & final words

The MySQL password you are asked to set on TurnKey LAMP (and LAMP based appliances) firstboot is for the "adminer" MySQL user account. That is a root like user too (so you can do DB administration via Adminer) but as it's available via a web interface, it requires a password.

I can't really speak to why Ubuntu have chosen to do things differently. As I noted above, AFAIK use of socket authentication is the default for both MariaDB v10.4.3+ & MySQL v5.7+. FWIW the MySQL v8.0 docs (under "Socket Peer-Credential Authentication") note:

[...] auth_socket authentication is well suited to server administration user accounts for which access must be tightly restricted.

I hope that sets your mind at ease...

I noted what I've done and linked to the git diffs in a comment on the tracker.

FWIW the main issue was that tkldev-setup had the GPG key hardcoded into it and we've rotated the keys for Buster. The updated script now clones the repos first and gets the key from the Buildtasks config. So future iterations should "just work" (so long as the key ID has been updated in buildtasks).

Unfortunately, because the script is contained within the overlay of the TKLDev build code, it won't be included until the next release. I'll probably need to do a RC2 build of TKLDev to fix that...

In the meantime, if you wanted to, you could download it like this:

URL=https://raw.githubusercontent.com/turnkeylinux-apps/tkldev/master/overlay
FILE=usr/local/sbin/tkldev-setup
wget $URL/$FILE -O /$FILE

It probably won't do much if you run it as you've already manually set it up. But if you wanted to test it out, you could remove the unpacked bootstrap and test that part (it will automatically pull the latest commits for all the repos before it checks the bootstrap downloads). Obviously, you'll need to download the updated tkldev-setup script first (as per above), then remove the Buster bootstrap and re-run it like this:

rm -r /turnkey/fab/bootstraps/buster
tkldev-setup

FWIW, here's what I got when I just re-ran it (after just removing the unpacked Buster bootstrap as per above):

INFO [tkldev-setup]: /turnkey/buildtasks exists, attempting update.
Already up to date.
INFO [tkldev-setup]: /turnkey/tklbam-profiles exists, attempting update.
Already up to date.
INFO [tkldev-setup]: /turnkey/fab/cdroots exists, attempting update.
Already up to date.
INFO [tkldev-setup]: /turnkey/fab/common exists, attempting update.
Already up to date.
INFO [tkldev-setup]: /turnkey/fab/products/core exists, attempting update.
Already up to date.
INFO [tkldev-setup]: Downloading bootstrap-buster-amd64
File 'bootstrap-buster-amd64.tar.gz' already there; not retrieving.

File 'bootstrap-buster-amd64.tar.gz.hash' already there; not retrieving.

INFO [signature-verify]: Verifying GPG signature
gpg: Signature made Sun Mar  8 10:37:48 2020 UTC
gpg:                using RSA key F190A48B54DC56B2C7F24DCBAC5EB00493E5BC1C
gpg: Good signature from "TurnKey GNU/Linux Buster Images (GPG signing key for TurnKey Linux Buster Images) " [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: A8B2 EF42 8781 9B03 D351  6CCA 7623 1C20 425E 9772
     Subkey fingerprint: F190 A48B 54DC 56B2 C7F2  4DCB AC5E B004 93E5 BC1C
INFO [signature-verify]: GPG verification success.
INFO [signature-verify]: Verifying checksum.
INFO [signature-verify]: Checksum verification success.
INFO [tkldev-setup]: Unpacking bootstrap-buster-amd64
INFO [tkldev-setup]: tkldev-setup complete.

Thanks to your feedback Ian, we've just published an updated v16.0rc2 TKLDev. Hopefully it should all "just work" now! :)

Great work, thanks so much Ian! :)

I'm currently working my way through some bugs that I've discovered in other build formats (e.g. OVA and LXC). As soon as I have all the other builds working, I'll start releasing v16.0 stable appliances.

Thanks again, your help is invaluable.

I only just noticed this post (for some reason I only get notifications for forum posts?!).

Anyway, I'm really hoping to have v16.0 LAMP and a few others released this week. However, we've had some internal infrastructure issues and I've been diverted to deal with that as a matter of urgency. I'm still hoping to get them out this week, but I can't promise anything...