v16.0 Stable Release #1 - 10 x ISOs, Hub & Proxmox/LXC builds

Thanks for sharing your experience. As both yourself and doxford (below) note, it looks like this bug still exists in Proxmox. :(

Also, I noted the workaround noted on the Proxmox forums/bug in the issue description but I didn't explicitly note it here. So here it is:

a reliable workaround appears to be enabling "Nesting" for the privileged container via Container -> Options -> Features -> Nesting (source: Proxmox forum thread). Note that there are security implications to this workaround (e.g. exposing the hosts /proc & /sys as read/write) so where possible, running a unprivileged container is preferable.

Unfortunately, I don't have more time to dig in deeper on this privileged container on Proxmox issue right now. Once I have published more of the library (and I've upgraded my local Proxmox server), I'll revisit this issue and see what we might be able to do on our end to improve things and/or push Proxmox to see what they can do.

If you'd like to dig in further yourself, there are some threads on the Proxmox forums that discuss ideas on alternate workarounds.

Thanks too for your report re MySQL/MariaDB password. I've opened an issue for that now too.

I'm glad to hear that you find the confconsole useful! :)

As you and peerx (bove) note, this issue does indeed still seem to affect PVE v6.x :(

As I just posted, when I get a chance, I'll dig in a bit deeper and see what else might be possible. In the meantime I also posted a workaround above, or probably best to just use unprivileged containers unless you need privileged ones.

FWIW I can confirm the issue using our v16.0 LAMP template in a privileged container on my local Proxmox server (Proxmox v5.x - based on Debian 9/Stretch).

I'd be really interested to know what Proxmox version you are running? If you are unsure, you can see it in webUI (towards the top right, to the left of the Promox logo) or by running 'pveversion'. I'd be particularly interested if it's still an issue in Proxmox v6.x.

In the meantime I've opened an issue on our tracker. I've written up details there, but I'll reiterate them here for convenience.

Firstly, apologies that I missed that. TBH, I probably should have double checked privileged containers. As it was, I only tested unprivileged ones as I mistakenly assumed that if unprivileged ones worked, then privileged containers would "just work" too. Obviously not...

So digging a little deeper, the issue appears to be a combination of LXC on the host and/or SystemD (and perhaps AppAmour too?!) on the guest. Although it primarily appears to be considered a LXC bug (at least by Debian).

Note that there is a Debian LXC bug - closed by patches which were applied to LXC to resolve this behaviour. I just checked the Debian 10/Buster LXC package to confirm that the version from the repos is fixed. As it was fixed in Debian some time ago (and is included in Buster repos OOTB) I assume that newer (i.e. v6.x / Debian 10/Buster based) Proxmox versions would not have this bug either?! They provide their own LXC package, but I assume it would likely either be running a newer version; or they would have applied the patches themselves?

Note that there is a (still open) Proxmox bug applying to Proxmox v5.x, so my guess is that this likely only applies to the older Proxmox version?! As I noted on the issue I posted though, that may not be the case?!

If anyone running Proxmox v6.x can test and confirm either way, I'd especially love to hear.

Yes we fixed them so that they should now launch as unprivileged containers. A few may still not work properly unprivileged due to specific software limitations (e.g. the Domain Controller), but most should be fine.

But are you saying that you are now having issues running our containers as privileged? They should still run fine like that?! If I understand correctly, please provide more info so I can look into it.

Yeah it's been a monumental slog, but we're on the home stretch now. And I feel fairly confident that some of the work that has been done (particularly behind-the-scenes infrastructure stuff) will actually make things a bit smoother into the future. We'll see I guess...

Anyway to answer your question, I'm awaiting news from AWS Marketplace team whether we can do it programatically or not. If we can get that up and running within the next week or 2, we'll update them via that (and all future updates will happen much quicker than they have in the past, I would guess probably within a day of Hub updates). If that takes more than a week or 2, I'll do them manually (like we always have done historically)... FWIW the manual updates take me a bit of time to prepare on our end, then can take up to a week (occasionally more) for AWS to process (they process them manually on their end too). So fingers crossed we'll be able to (semi-)automate them. It'll make it quicker and easier for everyone...

I didn't even bother reading it (I'm an English speaking monoglot) but I have no idea... Regardless, it's gone! :)

<<OFF TOPIC>>

And why the Vietnamese COVID spam?

Somehow they don't have many cases but having been to Saigon in January and seen a "wet market" I am not surprised these are considered the source. Negligible hygiene but maybe they all acquire immunity by repeated exposure. Hoping I acquired some!

It looks like I missed this post while I was writing mine...

Hopefully your questions are answered in that? As I noted, whilst it may seem a little "hidden" to you (the way that you generally use TurnKey), it should actually be "more obvious" to other users on other platforms (e.g. those that only access via SSH).

As I think I may have noted, the reason for only launching it on the first log in, is because I anticipate that it's most valuable to users on their first log in (or at least early within the lifetime of their server).

I spent a lot of time thinking, discussing and pondering how to best do this, trying to see it through the eyes of lots of different users in different usage scenarios. So I don't want to rush into changing things again. However, in the longer term, I'm not particularly attached to how it is now (just don't want to rush into changing things again until I've got some more feedback). So I am certainly open to further conversation on how it "should" work. So please feel free to share more of your thoughts.

Further checking.

LAMP15 boots to a screen with "Advanced Menu" available. SIMPLE

LAMP16 boots to a screen with just "Quit" option. That requires logging in and then the screen shows the option of an "Advanced Menu". However at subsequent boots, "Quit" option, then logging in just leaves you at the command line. There is mention of "For Advanced commandline config run: confconsole".Running that gives the screen with "Advanced Menu"!! Version 15 was so much easier. Why have a menu that just offers the option to Quit?

And NextCloud_16 is the same.

So the functionality is there but rather hidden.

I actually missed that bit of Ian's post. Thanks for highlighting it for me!

The suggestion of a "Shutdown" option makes a ton of sense! Thanks for explicitly suggesting it!

I might even try to add this sooner rather than later, but regardless, I've opened an issue so it doesn't get forgotten.

Hi Jeremy,

IanH has already mentioned in this post: In order to softly shutdown an appliance, now you have to loggin in, and run the "confconsole" command or the "shutdown" command.

Instead of having only the "Quit" option in the initial TurnKey Configuration Console, should it be possible to have both the "Quit" and the "Shutdown" options?

Does it make sense?

[Oops, I posted this in the wrong place... This is a reply to this post.]

On first log in (even if via SSH), you should be met with the "complete" Confconsole. After that (as noted in the MOTD; message of the day) for future usage, you can run it from the commandline via:

confconsole

It is also possible to make it run every login if you wish (but we figured the first time, with a MOTD reminder, is probably enough for most users). I still need to update the docs, but it's the "autostart " option within the Confconsole config file (/etc/confconsole/confconsole.conf). It can be "true" or "false" (the implications of whcih should be obvious). The other option is "once" which means it will run once at next log in.

TBH, I probably should have made specific note about this change, so let me do it here for now. I might move this up into the blog post, or perhaps in a new post, but for now, I'll just leave it here...

The primary reason that we changed it is that systemd (the Linux "init" system aka daemon manager) is getting really good at starting and running services asynchronously. That's great for improving boot speed, etc. But it also means that on firstboot, inithooks and confconsole were fighting for control of the terminal. That was a big issue on firstboot, but might also be an issue for later boots too (inithooks actually runs on everyboot). After spending a ton of time trying to get that working nicely, I did work out a way that we could make it work. But by then, I had also been trying to think about the issue laterally; identifying the problem we were trying to resolve and what new issues our solution may have been creating?!

So whilst I resolved the issue on how to make them start in the desired order, I also questioned why we were running Confconsole as a service anyway?! IMO there is no fundamental reason why it should be, and leveraging inithooks to launch it (when it was completed) seemed sensible. But then I realised that there was probably value in not allowing system config as root without log in. Plus, after spending so much time and energy on improving it (under the hood) it was also potentially of value to ensure that users are aware of Confconsole (at least see it once - on first login). Users who only use headless TurnKey deployment (e.g. only access via SSH, Webshell & Webmin), many users may not have even noticed that confconsole exists!

It's probably not unreasonable to allow Confconsole to run (in "complete" mode) following the firstboot questions. But to support that and allow it to also appear on first login, then that may get a bit tiresome for those users with direct terminal access. FWIW, I did also investigate leveraging PAM to allow access to the "Advanced" menu of Confconsole within Confconsole itself (i.e. log in via Confconsole rather than being dumped to the console), but I wasn't really happy with the Python PAM options.

So whilst it sounds like it comes as a bit of a rude shock to you, I hope overall, you'll agree that the new way of doing things is ok. I'd love to hear further feedback and thoughts, both form you and others...

Great work on getting this out BUT what happened to the menu allowing IP address change, shut down, etc?

All I see on LAMP is "Quit" option which drops me to the command line :-(  

That is typical of the CORE install.

OK, it was my first install of the LAMP 16.0 ISO on ESXi so I'll try again to check it out but there are few options I could have done wrong!

I have just finished publishing the next batch and that includes the PostgreSQL app! ;).

The 3rd batch are already in the pipeline, so it's unlikely that Mattermost will make it into that one, but we'll see what we can do re pushing that up the priority list.

Re TKLDev, we'd love to add some new apps, so any dev you want to share would be warmly welcomed. Also there have been a few TKLDev changes in v16.0, although hopefully nothing that will upset you too much! :) I still need to update the docs, but they should mostly be ok. As per always, please ask if you have questions.

Thank you for all the hard work!!

Would dearly love to see a Mattermost build... maybe easy since it's a Go blob like Gitea? Also a straight PostgreSQL build <3

I'm hoping to finally get some time to look into the builds I want to make now that 16.x is shipping - got a lot more going on in my business life that might converge into relearning TKLdev and hopefully creating some recipes :)

Sorry to be cheeky... :)

On a more serious note, the v15.x LXC appliance only provides support for v15.x (and theoretically earlier) appliances. So it not working for v16.x guests is somewhat expected.

I have made a bit of a start on the v16.0 LXC appliance but not ETA on when it will be done. I will aim to preempt v17.x as much as I can, but it's hard to know what might be required for a future major release.

In theory, once we've updated and released the v16.0 LXC appliance you could manually download the relevant updated files if you wish and it would likely work. Although it is possible that you'd need to update to the new v16.0 release to proerly run v16.x appliances.

[EDIT] actually, after posting, I just re-read your post. It seems that the mechanism that we included in v15.x to ensure that LXC always download the latest guest appliance has had some unintended consequence. Thanks for reporting, I'll aim to keep that in mind when I do the v16.0 update.

Perhaps we should make it default to the most recent release of the same major version as the LXC host? I.e. so your LXC host would default to v15.2 Nextcloud & v15.5 Drupal8 (and would only try v16.x if you explicitly asked it to).

Or perhaps there should be a switch to specify major version?

I'd be really interested in your thoughts!

Maybe I'm doing something wrong, but I've tried on several Turnkey LXC installations, and a couple more on bare metal and VMs just to be sure, but I'm unable to install/run any v16 container in the Turnkey LXC host, in all cases I get:

"FATAL [lxc-turnkey]: turnkey version 16.0-buster is not recognized:
lxc-create: lxccontainer.c: create_run_template: 1297 container creation template for nextcloud failed
lxc-create: tools/lxc_create.c: main: 318 Error creating container nextcloud"

If I manually specify the proper version 15 (f.e: -v 15.2-stretch) it works flawlessly, I have tried many others just to be sure it is not a nextcloud problem at all.

Am I missing something here?