Upcoming TurnKey v17.0/Bullseye apt repo changes - Bullseye users may need to manually clear caches

Status: [complete - update pushed]

This is a timely reminder that until we release a new major version as "stable", any assets and infrastructure associated with that upcoming release can't be considered stable either... v17.0 development is still ongoing and we're getting close to pushing out the first batch of updated v17.0 appliances. But before we do, we have some infrastructure updates to do - which may effect some early adopting users.

Infrastructure updates

Please note that this does not apply to any previous releases, up to and including the current stable v16.x releases (mostly v16.1; but a few are v16.2 or perhaps higher). This only applies to those testing/using the v17.0 release candidates and/or any users who have done an "in place" Debian upgrade to a Bullseye base.

As part of the move to a Debian Bullseye base, we've also been doing some infrastructure updates. This includes porting the python scripts that we use for packaging and managing our apt repos to python3 (many were still using python2). As part of that, we have hit a few minor issues. Whilst I knew that it was possible that these changes might make subtle changes to our repository, I didn't expect that even just pushing to 'bullseye-testing' might have unexpected consequences for Bullseye 'main' repo! So a couple of weeks ago, I unintentionally broke our bullseye apt repos.

I also noticed that it affected my (v17.0rc1) TKLDev and some others also reached out and asked about/reported it. As an immediate workaround I re-pushed the original/previous content. That immediately resolved it for everyone (at least as far as I'm aware). But I still need to push those changes! I have just had to do my testing in a completely separate repository (I had hoped to do it with 'bullseye-testing', but wasn't able to without causing disruption). Now the "proper" roll out is about to occur, it will likely have impact for anyone using any of the TurnKey Linux 'bullseye' repos (i.e. 'bullseye' aka 'bullseye-main' as well as 'bullseye-testing' and likely 'bullseye-security' too).

So even though technically no one should yet be relying on our 'bullseye' apt repo on a production server, I'm making this announcement now to give users a heads up! Hopefully this will minimise disruption for anyone affected. The first line (where it says "Status:") will be updated once I start the maintenance and again once it is completed.

Upcoming Linux Bullseye apt repo infrastructure updates - may require manual intervention

Prior to the stable release, I will need to push to our Bullseye repo again. I plan to do that early this week. I don't have an exact time, but I expect it will be (my timezone) morning of Tuesday, 1st February 2022 (i.e. this coming Tue). My current timezone is AEDT (UTC+11). So that will mean Monday afternoon/evening for many of you!

As I noted above, this will have no impact for users of TurnKey v16.x or earlier. It will almost certainly impact anybody using our Bullseye apt repo. Be that via running our RCs - or following an "in place" Debian upgrade (to a Debian Bullseye/11 base OS).

As noted above, once I have done the maintenance, the first line of this post (where it says "Status:") will be updated. (Probably to "Status: Done" or similar).

Easy workaround - clear all apt caches

Not sure if you're affected?

If you are not sure if you are affected or not, chances are you're not! But if you want to be 100% sure, please run the following command:

grep -R bullseye /etc/apt/sources.list.d

If that returns any results, then you will likely be affected. Even then though, if running 'apt update' works ok with no errors or warning, then you still should be good. Worst case scenario, running the below commands won't do any damage.

If you have any concerns and/or are at all confused, then please post a comment below describing what you ae seeing and I'll do my best to help you out.

Core 17.0rc1 & other apps updated to Debian Bullseye base

If you are running Core 17.0rc1 - or any of our appliances that have been upgraded to a Debian Bullseye/11 base OS (with the exclusion of TKLDev) then the below should be plenty enough to get things all cleared up:

apt clean
find /var/lib/apt/lists/ -type f -exec rm {} +
apt update

FYI, the first line will remove all cached .deb package files. The second will find and delete all the remote apt repository package lists and other related files (only the TurnKey ones will be affected, but it's much easier to just remove them all, then update them all again). Finally, (re)update the remote apt repository package lists and other related files.

TKLDev v17.0rc1

Beyond the above step(s) (to clear the local apt cache) on TKLDev, you'll also need to clear the Squid cache. Squid is used as the new internet caching proxy service for TKLDev downloads in v17.0. The old local caching service we used to use in TKLDev (Polipo) is no longer packaged in Debian, so you likely no longer have that installed, even if you have upgraded a v16.x TKLDev. If you have somehow updated to a Bullseye base OS but still have Polipo, I suggest creating a new TKLDev v17.0rc1 VM and use that instead.

Note that because the TKLDev host system also uses the Squid cache itself, you'll need to clear the Squid cache first, then clear the local apt cache (as noted above).

To clear the Squid cache (both in memory and on disk), run the following commands:

systemctl stop squid.service # may take a while to stop...
find /var/spool/squid -maxdepth 1 -type d -not -path "*/squid" -not -path "*/ssl_db" -exec rm -rf {} +
find /var/spool/squid -maxdepth 1 -type f -name "*state*" -exec rm {} +
systemctl start squid.service

In summary (aka TL; DR)

If you are using TurnKey with a Debian Bullseye/11 base OS, please be sure to clear your apt caches - after I push updates. Check the first line of this post to see whether the updates have been applied yet or not.

If you experience any issues which you think may be related; please comment below, giving as much detail as possible.


Jeremy Davis's picture

It's later than I had intended, but about to push updates now. I'll be back ASAP to notify about completion of the update.

Jeremy Davis's picture

It took a bit longer than I had hoped, but I'm done now.

Please follow the instructions posted in the above blog post. If you still experience issues after that, please post below. Please provide as much info as possible.

L. Arnold's picture

Trying TKLDEV 17 RC!.  did not get errors but tried a "Core... Make" after and would not clear.
did work before.  (did this to Linode, but trying to get something besides an ISO build)

Also none of the ./bt (various) would work with TKLDEV 17 RC!.

Out of the box, I could do a "Core/Make to create product.iso in the Core Builds folder.

Jeremy Davis's picture

FYI, the issue noted in this blog post only applied to users who were already using v17.x (or v16.x upgraded to Debian 11/Bullseye base) before I made the apt changes. So if your server wasn't running before 1st Feb, then this issue does not apply!

Having said that, clearing the caches as noted in this blog post shouldn't do any harm. Although on TKLDev, you need to clear the squid cache first, then clear the apt cache.

As to the issues you are hitting when running the 'bt-' scripts, AFAIK, they should be working ok?! If you could provide some more info about the error messages you are getting that would be useful.

L. Arnold's picture

working through trying to get a BuildTask to work w/ TKLDEV 17.0 rc1
tried:  ./bt-otc tkldev-17.0rc1-bullseye-amd64

I did reinstall to a larger Drive w/ ISO in Linode

looked promising then got.  Perhaps I shold work through the above again. 

FATAL [parse-appname-version] unrecognized codename: bullseye


L. Arnold's picture

I've generated a ISO (product.iso) using TKLDEV 1rc1 but for the life of me I cannot get it to boot to buil a new system the way I can from Mirror pulled ISOs.

I am trying to use these as RAW Disk images converted into a new Drive Image with DD command.  That shows it is working, but not booting.

Probably need to generate the same via TKL 16 for comparison.  Definitely with the Mirror images I pull them in with the CURL command, while, posting the ISO into Dropbox I can only download it (off the web) with a wget command.

Work around was to use DD on my running TKLDEV to a a new Drive on the Cloud then try to boot from that new drive...  Basically not find it's way it seems.  

Trying some more next.

Jeremy Davis's picture

There are some changes to how the ISO boots in v17.0. I started implementing UEFI support. The UEFI support is in 2 stages, the first is to build an ISO that will boot via UEFI (which has been done) and the other is installer (particularly partitioning) support for UEFI (incomplete and run out of time for v17.0).

In theory, it should still be a "hybrid" ISO, so writing it out via 'dd' should still work fine. Having said that, I haven't actually tested that, and your experience suggests that it isn't working as it should. So I will test myself today.

FWIW, as I hinted above, I've run out of time to finish the UEFI support (in the installer) for v17.0. So worst case scenario, if the UEFI boot support has broken the ease of writing via 'dd', then I'll just remove those changes for v17.0 (and resolve the issue when I implement it with installer support).

Thanks again for the report. If I manage to get the 'dd' working ok, I'll post instructions.

Jeremy Davis's picture

I removed the double post.

I should have read a bit more before I responded. That error message is very useful. I've now fixed the script you note. It should now support 'bullseye' too!

You can see the changes I made here. And you can pull them into your TKLDev (assuming you haven't made any changes to common's git config), like this:

cd common
git pull origin master


Add new comment