Do you love TurnKey? Have a few spare cycles and want to help out?

2022-05-17 Update: Please note that due to a bug, all v17.0 appliances have now been bugfixed and rebuilt as v17.1. The v17.x release will continue, with all new appliances to be released as v17.1.
Important note: As noted in the v17.1 release announcement the root issue that caused the kernel panic bug still exists in TKLDev! This DOES NOT affect building alternate builds from "official" v17.1 apps, but will affect any ISOs that you build with TKLDev v17.0 or v17.1. The bugfix/v17.1 announcement includes (temporary) code you can add to the conf.d/main script to resolve the issue. We're still working on a proper fix. Once it's done, we'll release a new version of 'fab', and a new release of TKLDev. Existing v17.x TKLDev deployments will be able to install the bugfixed version of 'fab'. More info as it comes to hand.

So do you love TurnKey? Or even just like it quite a bit?! And have some time and energy and a desire to help out? If so, have I got a job for you! :)

As many of you would be aware, we're still plugging away at the well overdue v17.0 release - if you're not, please check out the most recent announcement (at the time of writing). So far, we've only produced 25, but lots more are well on the way behind the scenes. Unfortunately, our resources are limited, so we're prioritising ISO and Hub builds for now. That means that a number of our "optimized" or "alternate" buildtypes will not be available until we can circle back to them. As the buildcode for the different buildtypes is updated for v17.0, we can batch process all the v17.0 ISOs that are available.

Buildtypes remaining

I'll try to keep this list below updated but as of the time of writing this is the status of the optimized/alternate buildtypes:

2022-05-11 - Proxmox/LXC - updated code merged into master.

  • Hub - working and being progressively published
  • Proxmox/LXC - merged into master - working but needs more testing
  • VM (VMDK & OVA) - untested
  • OpenStack (KVM) - untested
  • Xen (HVM) - untested
  • Docker - untested

This is where you can come in

So if your favourite buildtype (above) isn't yet available for v17.0, here's how you can help push it forward. You'll be helping yourself, but also helping TurnKey and all other users.

What if I can't code? What if I don't know git or other weird commands?

No coding experience required. If you've run commands on a Linux terminal (and can google), then that's enough to get started! Most of the build code is bash script, which is essentially just a heap of commands in a file. Plus files can be overlaid - but let's not dig into that too much just yet.

I'm happy to coach (in the comments below). Even if you just copy/paste the relevant commands and report back with any error messages - or stories of success, that is extremely useful. Just doing that much cuts down the time it takes us in development and testing, which means the sooner they'll be available for everyone.

What you'll need

You'll need access to the environment that the particular build requires, although I guess that's probably both obvious and likely.

You'll also need to install TKLDev as a VM. One easy way is to just use VirtualBox (or similar) on a desktop. See our VirtualBox install tutorial for details (install from v17.0 TKLDev ISO). Once you have TKLDev installed and running, you could build ISOs - ours, or your own custom one. But that's not what I'm covering here. Instead, we'll look at converting an existing ISO to a different buildtype. This could be your own custom ISO, or one that we have already built, but I'll only focus on building from TurnKey ISOs (including downloading them).

Doing a build

First up, you'll need to move to buildtasks and do the simple setup. The default config should be fine to get started:

cd buildtasks
cp -R config.example config

Then do a build. For an untested build, I suggest starting with Core. Core is the most simple appliance with the least moving parts, so the most likely to work and/or need the least work. If the buildtype you wish to work on is already (reportedly) working, then testing other appliances is a probably better place to start. I'll use Core for my example, but please feel free to update the commands to build other appliances. I recommend only building v17.0 appliances at this point.

If you want to test the default ('master' branch) then you can just run the relevant script. E.g. to attempt a build of Core to VM (VMDK and OVA):

./bt-vm core-17.0-bullseye-amd64

How to pull in updated code for testing aka getting 'gitty'

Git micro primer

We use git for code version control. You don't need to know lots about it, but it's probably worth being aware of a few basic terms and their meaning:

  • Commit: a "commit" is the code at a point in time. It is a set of changes applied to the commit before it (it's "parent"). Each one has a "commit id" which is a SHA-1 hash. It also has a message. The current commit can also be referred to as 'HEAD'. A new "commit" occurs when a new change to the code is saved (via the "git commit" command).
  • Branch: a "branch" is a collection of one or more "commits". The default branch (at least for TurnKey projects) is usually 'master'. Development usually (and generally should) happens in alternate branch names. Descriptive branch names are recommended.
  • Repository: a "repository" (in the context of git) is a directory including a .git directory and whatever else is being stored in it.
  • Remote: a "remote" is a remote git repository. TurnKey uses GitHub, but there are lots of other options.
  • Fetch: 'fetching' is the process of updating the local store of info about a "remote".
  • Checkout: you can "checkout" an alternate "branch" (either local or remote, old or new).
  • Stash: "stash' updated code that hasn't yet been committed, particular useful if you want to try someone else's "branch".

Checkout a testing branch and do your build

Often during development, there may be updated code that is still in a development branch and has not yet been merged into master. If you wish to test updated code that is not yet merged, you'll need to fetch it and the do a checkout. I'll use the Proxmox/LXC update as an example. This example has now been merged into 'master', so will no longer work.

First, add my GitHub account as a remote and fetch my available branches and checkout my 'fix-1636' branch:

git remote add jeremy
git fetch jeremy
git checkout fix-1636

Now you can test my update for the Proxmox/LXC build for the new BookStack appliance:

./bt-container bookstack-17.0-bullseye-amd64

You'll find the result in the /mnt/builds/ sub-directory. E.g. for bt-container: /mnt/builds/container/

Let me know how you go.

I'm hoping to keep this post updated as we work through v17.0. So please share your experience in the comments below. I'll aim to update info as things change and I'll probably add further info/sections here, especially if there are common issues and/or troubleshooting info that will help others (without having ot read all hte comments). So please have a go and/or ask any questions you might have and I'll do my best to guide you.


Chris's picture

I've never done this kind of build before but I thought I'd have a go to see if I can help the process along a little. I've set up a privileged TKLDEV LXC container on Proxmox 7 (1TB disk, 16GB RAM, 8 cores) and then followed the instructions above to try to build a core 17.0 container using "./bt-container core-17.0-bullseye-amd64".

That downloads the ISO into /mnt/isos and then I get an error:

# extracting root filesystem and isolinux from ISO
mount: turnkey-core-17.0-bullseye-amd64.mount: mount failed: Operation not permitted.

I've tried setting the builds and isos directories in /mnt to 777 but that didn't help. I'm running as root so should have full privileges anyway. I've tried googling for an answer but nothing is showing up.

Any ideas on what I'm doing wrong?

Jeremy Davis's picture

Thanks for having a go! Sorry to hear you hit a hurdle straight out of the gates...

Regarding your error, the message says that it doesn't have permission to 'mount'. I suspect that there is something not right in the container config. Unfortunately, I'm not familiar with running it in a container (I run it in a full VM). A quick google suggests that it can be done but it sounds like a bit of a pain.

I recommend installing from ISO to a VM. That way you can build v17.x on v17.x too - which is better IMO. Unfortunately, there is a nasty bug in v17.0 so I've removed all the v17.0 links. But the v17.0 TKLDev ISO is still on the mirror if you want to use it.

Alternatively, I should have fixed ISOs (they'll be v17.1) ready for use within a few days.

Chris's picture

I think apparmor is blocking mount, even in a privileged container. I tried a few things to try to loosen apparmor's control but couldn't find the magic incantation. So...

I installed the TKLDEV 17.0 ISO as a VM, which took a bit of figuring out with network and DNS needing manual config after installation. The VMs are really best used with DHCP but I'm using a static IP config so I had no network initially, and that stops tkldev-setup from running. I figured it out in the end though.

I've now got a working TKLDEV installation and have managed to build a Core 17.0-01 container which works in Proxmox. Which leads to a couple more questions.

  1. I assume you will let us know when 17.1 is available for building from?
  2. What specifically are you looking for when we build something? Errors during the build? Warnings during the build? Errors/warnings when the container is installed and run? I know I saw a warning or two during the build I did. I don't know whether they're 'normal' so you might get some reports that aren't necessary, but I guess that's part of the learning process! I won't report what I saw, I'll wait to see if they still occur in v17.1 and report them then if they do.
  3. Is there any documentation that describes the build process? I'm clearly not an expert in building containers, but it would be interesting (yes I have a strange mind!) to understand the process better which might allow me to find the causes of errors/warnings and maybe even fixes for them.


Jeremy Davis's picture

Excellent, glad to hear that worked.

FWIW, it would be awesome to work out what is required to get TKLDev running in a container, but it isn't a current priority for us. If you do turn back to that at some point, please do share.

Thanks too for confirming that the Core LXC/Proxmox build works on your setup. That's a solid start. Now we know that works, it'd be good to check out some of the more complex appliances. Note that at this point, I would recommend avoiding our VPN apps and the Domain Controller.

In specific answer to your questions:

I assume you will let us know when 17.1 is available for building from?

I will publish a blog post about it, plus will cross post links elsewhere (including updating blog posts like this one). So hopefully you should get notified.

What specifically are you looking for when we build something? Errors during the build? Warnings during the build? Errors/warnings when the container is installed and run? I know I saw a warning or two during the build I did. I don't know whether they're 'normal' so you might get some reports that aren't necessary, but I guess that's part of the learning process! I won't report what I saw, I'll wait to see if they still occur in v17.1 and report them then if they do.

Showstopping errors during build are the first thing to address. All critical errors will halt the build process. In the case of the Proxmox/LXC builds, we've already passed that point. I haven't even tried any of the other builds, so I have no idea what shape they are in.

Once you have the build completing successfully, it is worth having a quick look over the build output. As you noted, there will likely be some warnings during the build process. Most (if not all) of them are likely not really an issue, but it's worth double checking. It's especially important to note new warnings during a transition (i.e. moving to a newer Debian base - like we are for v17.x). Hence why documenting these would be useful for future people such as yourself.

The next step is to ensure that the build actually works. Again we've somewhat got to that point with the Proxmox/LXC builds. At least for Core. For what it's worth, initially they just locked up. But we've worked that issue out (and now remove acpid - which was causing the lockup).

Regardless, because it's still somewhat new, the more reports the better (even if that's just "works for me too"). There have been plenty of times that things work fine for me, but others experience issues. So the more people testing, the better the chance that we can find (and fix) any issues before release.

In the case of Core, I've personally only tested logging in via SSH and had a quick poke around. I checked resource usage (via htop - top is installed by default, but htop is better and can be installed via 'apt install -y htop') as well as quickly scanning the journal ('journalctl'). Within an unprivileged LXC instance, there'll likely be warnings and many (hopefully all) may be expected. But again, if you're not sure, they're often worth a look (or just report them so myself &/or others can).

Again documenting these would be a good idea. Also FWIW, from my quick tests, both Webshell and Webmin appeared to work ok, and I also briefly checked that our custom services and scripts (such as inithooks (firstboot scripts) and confconsole) appeared ok. Once these common components are confirmed working in Core, the need to devote too much more time to further testing these in other appliances is generally not required. Although many appliances have custom inithooks, so I always check /var/log/inithooks.log looking for any scripts that exit with a non-zero exit code (usually 'exit 1').

From there, as noted above, we need to test more complex appliances, that include different software and services. In the case of web apps, I will often check the main app via my web browser, enable "developer tools" and check for any 404 or javascript errors. (Again some are ok, like 404 for favicon.ico - others not!). As you're likely aware, a few of our appliances don't play nice in an unprivileged LXC instance. As noted, ones that come to mind are our VPN apps and our Domain Controller. About 70% of our appliances are based on LAMP, so testing a few of our webapps is pretty important. To be explicit, we do fairly simple "smoke tests" on all the ISO builds. So in theory, any general "showstopper" bugs you find should be specific to Proxmox/LXC build. Although maybe you've spotted something that we've missed?

As far as reporting goes, it's much better to get a report that doesn't matter, than not get a report when it is an issue! As with many tech related issues, there is generally never such a thing as too much information! Whilst it's hard to explicitly tell you want to report and what not too, if you report stuff that isn't important, I'll likely be able to filter that out pretty easy. But I can't read your mind to get more info about things you don't share! :)

Again, it's probably a good idea to document any specific warnings that aren't a problem. This could be an opportunity to start that process? We could then even consider suppressing or at least contextualising them in the future perhaps?!

As you learn more, you'll likely get better at knowing what's important to report and what specifically relevant info to share. But even then, I still tend to be fairly verbose when reporting issues. Actually, in my experience, just the process of sharing info can often lead to insights. If I had a dollar for everytime I've been able to resolve an issue myself, simply by going through the process of carefully and exhaustively documenting what is occurring - I'd be a wealthy man! :) Bottom line, please don't hold back on what you share.

Is there any documentation that describes the build process? I'm clearly not an expert in building containers, but it would be interesting (yes I have a strange mind!) to understand the process better which might allow me to find the causes of errors/warnings and maybe even fixes for them.

There is a ton of documentation on the build process but unfortunately, it's not been well maintained, nor is it organised all that well. Most of it relates to the appliance build code (i.e. building appliances from scratch to ISO). So may not be that directly relevant to you in this scenario. If you do have a read of any of that and notice any issues, anything that doesn't make sense, or doesn't match what you see, please do not hesitate to ask and/or report any issues or areas for improvement.

Unfortunately, there isn't much documentation specific to the "alternate" or "optimized" builds - as provided by buildtasks. The build scripts are mostly just bash (with a few python scripts within bin). They may look a bit intimidating if you're not familiar with bash, but they're actually relatively straight forward. They're more-or-less just a list of commands to run, with some control logic (which could, at least in theory, just be copy/pasted straight into the command line).

In the the case of most of the bt scripts, here's the basic flow of what they do:

  • unpack the iso and extract the root filesystem
  • remove and/or install specific packages
  • tweak config as relevant (much of it via "patches" - which are just bash scripts and/or files to overlay)
  • cleanup to remove any remaining cruft
  • pack updated root filesystem into relevant package/container - e.g. bt-container compresses to a tar ball ready to be used on Proxmox

I hope that provides some insight and value. If you have further questions, etc. Please ask.

Thanks again for helping out. Just having people such as yourself engaging and interested in learning and helping out is like fuel for us!

deutrino's picture

Hi there, I'll try to take a look at this if I have any spare cycles this coming week, and build the containers I use etc.

Jeremy Davis's picture

Would love your input! :)

deutrino's picture

Thinking I'll try to start with Core, Syncthing, the VPN appliances, the LAMP stack (and maybe others), and Wordpress, if testing would be useful for any of those.

Jeremy Davis's picture

TBH, I'm fairly confident that they'll be good. But it'd be great to have confirmation of that. With enough community feedback, I'll be happy to publish.

Timothy McConnor's picture

It could be really helpful. I will definitely try it.


Add new comment