proxmox 7.3 turnkey 17.1.1 core apt update fails with "files has unexpected size Mirror sync in progress? [IP: 54.83.15.172 80]

Thanks so much for reporting this issue! I'll look into it and get back to you ASAP.

FWIW - I couldn't reproduce it initially, but I've managed to so will follow up and post back.

In the meantime, if you want to install software, so long as you don't need to install any TurnKey specific packages, you can (temporarily) disable the TurnKey repo like this:

sed -i.bak "\|archive.turnkeylinux.org| s|^|#|" /etc/apt/sources.list.d/*sources.list

That will create backup files (same name with '.bak' appended) of /etc/apt/sources.list.d/sources.list & /etc/apt/sources.list.d/security-sources.list and comment out the turnkey lines in the original files.

To revert, just copy the backups back over the top of the commented files:

mv /etc/apt/sources.list.d/sources.list.bak /etc/apt/sources.list.d/sources.list
mv /etc/apt/sources.list.d/security.sources.list.bak /etc/apt/sources.list.d/security.sources.list

It should be fixed now - please check and confirm if you can. It'd be good to know 100% that I've fixed it.

It should be fixed now - please check and confirm if you can. It'd be good to know 100% that I've fixed it.

I just wanted to give you a quick update and apologise that it's not fixed yet.

Other than sleeping on it, I have spent most of the time since I first replied investigating what happened. Before I did anything, I needed to know the scope of what I'd potentially need to do. I wanted to be sure that I understood what had happened. It was especially important to be able to rule out something malicious (because I'd need to rotate keys if that was the case).

Turns out that the cause was an assumption that I made and the fact that I only tested on existing VM servers that were already running. I didn't test on a new instance and/or on LXC. That was an oversight and I deeply apologize.

Beyond investigation, yesterday I confirmed how I could recreate the issue on a server that wasn't previously experiencing it. I also set up a local apt repo test environment, so I will be able to minimize the chance of making things worse (I need to fix it for you guys, without breaking it for existing users). After sleeping on it, I have a plan!

I'll report back once I have something to share (hopefully good news...). Thanks for your patience.

Ok, so my testing indicates that it's now fixed. Can anyone who was having the issue please confirm that it no longer occurs? If you disabled the TurnKey repo, please be sure to re-enable that first. E.g.:

mv /etc/apt/sources.list.d/sources.list.bak /etc/apt/sources.list.d/sources.list
mv /etc/apt/sources.list.d/security.sources.list.bak /etc/apt/sources.list.d/security.sources.list

To test, all you need to do is run apt update, the output should now look like to this:

Hit:1 http://security.debian.org bullseye-security InRelease
Hit:2 http://deb.debian.org/debian unstable InRelease
Hit:3 http://deb.debian.org/debian bullseye InRelease
Ign:4 http://archive.turnkeylinux.org/debian bullseye-security InRelease
Ign:5 http://archive.turnkeylinux.org/debian bullseye InRelease
Ign:6 http://archive.turnkeylinux.org/debian bullseye-testing InRelease
Hit:7 http://archive.turnkeylinux.org/debian bullseye-security Release
Get:9 http://archive.turnkeylinux.org/debian bullseye Release [5659 B]
Hit:11 http://archive.turnkeylinux.org/debian bullseye-testing Release
Fetched 5659 B in 2s (2370 B/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

(Obviously yours may be a little different, but showing no errors or warnings)

root@porter ~# apt update
Hit:1 http://security.debian.org bullseye-security InRelease
Hit:2 http://deb.debian.org/debian bullseye InRelease
Hit:3 https://download.docker.com/linux/debian bullseye InRelease
Ign:4 http://archive.turnkeylinux.org/debian bullseye-security InRelease
Ign:5 http://archive.turnkeylinux.org/debian bullseye InRelease
Get:6 http://archive.turnkeylinux.org/debian bullseye-security Release [3863 B]
Get:7 http://archive.turnkeylinux.org/debian bullseye Release [5659 B]
Get:8 http://archive.turnkeylinux.org/debian bullseye-security Release.gpg [833 B]
Get:9 http://archive.turnkeylinux.org/debian bullseye Release.gpg [833 B]
Get:10 http://archive.turnkeylinux.org/debian bullseye/main amd64 Packages [34.0 kB]
Fetched 45.2 kB in 2s (26.6 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
45 packages can be upgraded. Run 'apt list --upgradable' to see them.

I thought that I would have enough time to get to it last week, but things got a bit crazy on Friday too. So I still haven't got it. Apologies about that.

I am about to have a look and hopefully it should be done really soon. I'll post back when I'm done.

Ok, it should be fixed now. Please confirm if you can (I'll assume no news is good news, but it's better if I know for sure).

Apologies again that it took me so long to fix.

Sorry but I've been called away so won't be able to fix this in the timeframe that I'd hoped (more-or-less immediately). I'll get back to it ASAP, possibly later today, or maybe tomorrow if I run out of day.

This post is perhaps a bit disjointed. Initially, I didn't understand the issue at all. But during the process of writing this post, it became very clear what was going on. So a lot of what I had started writing ended up being only vaguely relevant. So I've essentially rewritten it, with the last bit (once the cause dawned on me) moved to the start, and the original start, heavily edited - with what was left, added on the end (not immediately relevant, but gives context that may be relevant to you?).

TBH, I was initially confused by how the arch related to your issue and why you'd even mention your M1 chip? That was until I looked a bit closer at the apt output that you had posted previously! Under closer inspection, it was glaringly obvious! I noticed (emphasis obviously mine):

Get:7 http://archive.turnkeylinux.org/debian bullseye/main arm64 Packages [34.0 kB]
Err:7 http://archive.turnkeylinux.org/debian bullseye/main arm64 Packages

My previous casual perusal had missed the 'arm64' in there! Had I realized that sooner, I wouldn't have been so surprised by your experience!

I understand the issue now. It's because you're using the arm64 apt repo - which I didn't fix! - because I didn't realise that anyone was using it; hence didn't see a need to fix it! Despite the fact that we don't yet officially support ARM64, seeing as we already have packages there and the fix is super simple, I'll do that shortly and post back once it's done. So even though you won't yet be able to run TurnKey, you will at least be able to install TKLBAM!

If you try anything else from our ARM64 repo that doesn't install, work, or any other issue, please feel free to let me know. At this point I won't be able to prioritise it over immediate TurnKey stuff, but ideally I'd love all TurnKey software to be installable and working on any Debian/Ubuntu instance of a supported arch. (That's not yet the case, but it's my strong desire).

(This is the edited version of the original start to my post)

Currently TurnKey only officially supports AMD64 architecture (aka x86_64). We've been working towards supporting ARM (arm64/aarch64) too - which should work on M1 chips (the intention was specifically to support Raspberry Pi, but ideally we'd like to support all ARM64 chips). Some of the pieces are in place, but due to lack of resources and competing priorities, that effort has currently stalled. I do hope to circle back to it, but no idea on when I might get there... :(

As TurnKey only currently provides AMD64 builds, TurnKey not working on your Mac with an M1 chip is somewhat expected. AFAIK you can emulate ARM64 on AMD64 hardware, but not the other way around.

That looks like the same error as reported here. I wonder why the fix worked for me and at least one other, but not you?

I'm not sure if it'll work, but perhaps try clearing the local TurnKey apt repo lists and try again?

find /var/lib/apt/lists/ -type f -name "archive.turnkeylinux.org*" -exec rm {} +
apt update

Please let me know if that doesn't work and I'll dig in a little deeper.

Hit:1 http://deb.debian.org/debian bullseye InRelease
Hit:2 http://security.debian.org/debian-security bullseye-security InRelease
Hit:3 http://deb.debian.org/debian bullseye-updates InRelease  
Ign:4 http://archive.turnkeylinux.org/debian bullseye InRelease
Get:5 http://archive.turnkeylinux.org/debian bullseye Release [5,659 B]
Get:6 http://archive.turnkeylinux.org/debian bullseye Release.gpg [833 B]
Get:7 http://archive.turnkeylinux.org/debian bullseye/main arm64 Packages [34.0 kB]
Err:7 http://archive.turnkeylinux.org/debian bullseye/main arm64 Packages
File has unexpected size (34074 != 34030). Mirror sync in progress? [IP: 54.83.15.172 80]
  Hashes of expected file:
   - Filesize:34030 [weak]
   - SHA512:480a9ba7afd53bbf8cb22aa216543e81d93eda09cf24ff8e7658b2bc9a461a6d62ab81a6d4ed71d4014480cbfc2d32eb7959aa43a83b8297b5083b2dc0a6861c
   - SHA256:4e1a3afb495ce060073d40eb68158bc07a52e18ad3b929fd517101ca900ce31d
   - SHA1:4814cb4c75559f758422a39540af04be0136d715 [weak]
   - MD5Sum:cb2e4f8dee8183fd81aa91bd80b9f4f0 [weak]
  Release file created at: Wed, 02 Nov 2022 06:42:43 +0000

Great news! Thanks for the confirmation that it's working again. Thanks too for your patience while it wasn't! :)

[edit] And actually, you deserve a special mention for perseverance! Thank you so much for posting back after I accidentally removed your original post in my bleary-eyed early morning spam cleaning frenzy!

its fixed now. Thank you!