Upcoming TurnKey v17.0/Bullseye apt repo changes - Bullseye users may need to manually clear caches

Status: [complete - update pushed]

This is a timely reminder that until we release a new major version as "stable", any assets and infrastructure associated with that upcoming release can't be considered stable either... v17.0 development is still ongoing and we're getting close to pushing out the first batch of updated v17.0 appliances. But before we do, we have some infrastructure updates to do - which may effect some early adopting users.

Infrastructure updates

Please note that this does not apply to any previous releases, up to and including the current stable v16.x releases (mostly v16.1; but a few are v16.2 or perhaps higher). This only applies to those testing/using the v17.0 release candidates and/or any users who have done an "in place" Debian upgrade to a Bullseye base.

As part of the move to a Debian Bullseye base, we've also been doing some infrastructure updates. This includes porting the python scripts that we use for packaging and managing our apt repos to python3 (many were still using python2). As part of that, we have hit a few minor issues. Whilst I knew that it was possible that these changes might make subtle changes to our repository, I didn't expect that even just pushing to 'bullseye-testing' might have unexpected consequences for Bullseye 'main' repo! So a couple of weeks ago, I unintentionally broke our bullseye apt repos.

I also noticed that it affected my (v17.0rc1) TKLDev and some others also reached out and asked about/reported it. As an immediate workaround I re-pushed the original/previous content. That immediately resolved it for everyone (at least as far as I'm aware). But I still need to push those changes! I have just had to do my testing in a completely separate repository (I had hoped to do it with 'bullseye-testing', but wasn't able to without causing disruption). Now the "proper" roll out is about to occur, it will likely have impact for anyone using any of the TurnKey Linux 'bullseye' repos (i.e. 'bullseye' aka 'bullseye-main' as well as 'bullseye-testing' and likely 'bullseye-security' too).

So even though technically no one should yet be relying on our 'bullseye' apt repo on a production server, I'm making this announcement now to give users a heads up! Hopefully this will minimise disruption for anyone affected. The first line (where it says "Status:") will be updated once I start the maintenance and again once it is completed.

Upcoming Linux Bullseye apt repo infrastructure updates - may require manual intervention

Prior to the stable release, I will need to push to our Bullseye repo again. I plan to do that early this week. I don't have an exact time, but I expect it will be (my timezone) morning of Tuesday, 1st February 2022 (i.e. this coming Tue). My current timezone is AEDT (UTC+11). So that will mean Monday afternoon/evening for many of you!

As I noted above, this will have no impact for users of TurnKey v16.x or earlier. It will almost certainly impact anybody using our Bullseye apt repo. Be that via running our RCs - or following an "in place" Debian upgrade (to a Debian Bullseye/11 base OS).

As noted above, once I have done the maintenance, the first line of this post (where it says "Status:") will be updated. (Probably to "Status: Done" or similar).

Easy workaround - clear all apt caches

Not sure if you're affected?

If you are not sure if you are affected or not, chances are you're not! But if you want to be 100% sure, please run the following command:

grep -R bullseye /etc/apt/sources.list.d

If that returns any results, then you will likely be affected. Even then though, if running 'apt update' works ok with no errors or warning, then you still should be good. Worst case scenario, running the below commands won't do any damage.

If you have any concerns and/or are at all confused, then please post a comment below describing what you ae seeing and I'll do my best to help you out.

Core 17.0rc1 & other apps updated to Debian Bullseye base

If you are running Core 17.0rc1 - or any of our appliances that have been upgraded to a Debian Bullseye/11 base OS (with the exclusion of TKLDev) then the below should be plenty enough to get things all cleared up:

apt clean
find /var/lib/apt/lists/ -type f -exec rm {} +
apt update

FYI, the first line will remove all cached .deb package files. The second will find and delete all the remote apt repository package lists and other related files (only the TurnKey ones will be affected, but it's much easier to just remove them all, then update them all again). Finally, (re)update the remote apt repository package lists and other related files.

TKLDev v17.0rc1

Beyond the above step(s) (to clear the local apt cache) on TKLDev, you'll also need to clear the Squid cache. Squid is used as the new internet caching proxy service for TKLDev downloads in v17.0. The old local caching service we used to use in TKLDev (Polipo) is no longer packaged in Debian, so you likely no longer have that installed, even if you have upgraded a v16.x TKLDev. If you have somehow updated to a Bullseye base OS but still have Polipo, I suggest creating a new TKLDev v17.0rc1 VM and use that instead.

Note that because the TKLDev host system also uses the Squid cache itself, you'll need to clear the Squid cache first, then clear the local apt cache (as noted above).

To clear the Squid cache (both in memory and on disk), run the following commands:

systemctl stop squid.service # may take a while to stop...
find /var/spool/squid -maxdepth 1 -type d -not -path "*/squid" -not -path "*/ssl_db" -exec rm -rf {} +
find /var/spool/squid -maxdepth 1 -type f -name "*state*" -exec rm {} +
systemctl start squid.service

In summary (aka TL; DR)

If you are using TurnKey with a Debian Bullseye/11 base OS, please be sure to clear your apt caches - after I push updates. Check the first line of this post to see whether the updates have been applied yet or not.

If you experience any issues which you think may be related; please comment below, giving as much detail as possible.