TKLDev 17.1 coexistence with squid_deb_proxy_client

It sounds like you're on to it, but to ensure that we're on the same page, on TKLDev squid is set up as a caching proxy for http and https. We use squid with a more generic proxy config to allow us to cache all sorts of packages and/or files (not just apt). Apt on TKLDev should be using it's own squid cache - configured in /etc/apt/apt.conf.d/00proxy:

Acquire::http::proxy "http://127.0.0.1:3128";
Acquire::https::proxy "http://127.0.0.1:3128";

Beyond that, TBH, I'm not completely sure of the best way to achieve your ends.

Possibly the quickest/easiest way to go would be to reconfigure your TKLDev and the TKLDev build process to use your existing apt-cacher for fetching packages. Configuring TKLDev itself is easy (i.e. replace /etc/apt/apt.conf.d/00proxy). To adjust the building appliance's apt cache, adjust the common bootstrap_apt conf script (should locally be found at /turnkey/fab/common/conf/bootstrap_apt). You'll need to adjust the following lines that start with 'Aquire' (lines 38 & 39 currently):

# Configure apt proxy
cat > $CONF_DIR/01proxy <<EOF
Acquire::http::proxy "http://127.0.0.1:$PROXY_PORT";
Acquire::https::proxy "http://127.0.0.1:$PROXY_PORT";
EOF

Note that the squid proxy will still be running/available, so you can use it during builds if you want (e.g. caching a downloaded file). Different tools have different ways of setting a proxy, although most Linux utils and programs should honor http_proxy and https_proxy if you set them. Note that the env vars FAB_HTTP_PROXY & FAB_HTTPS_PROXY should contain the default relevant proxy url. Simply export the env vars to enable proxy caching for most commands/services. Do that (in your conf script) like this:

export http_proxy=$FAB_HTTP_PROXY
export https_proxy=$FAB_HTTPS_PROXY

As a longer term plan though, perhaps it's worth considering setting up your own local dedicated web caching server?! You could essentially re-implement the Squid setup we use on TKLDev, on a Core instance? You could then use it to replace your apt-cacher set up, as well as use it as a general caching web proxy to improve your web browsing and other online experience?!

Keep in mind though, that the way we have squid set up on TKLDev, it's essentially a MITM (man in the middle). It decrypts https web traffic, caches it and then re-encrypts it to your browser. So, you'll need to import the CA key into your OS and/or browser. It does also mean that https traffic which you'd generally assume is private, is actually being stored on your network. I didn't bother looking into it, but I'd assume that you could blacklist caching of specific sites (e.g. internet banking) so that connect directly to those sites that it makes the most sense.

Apologies that I didn't explicitly respond to your question about squid-deb-proxy-client. Assuming it's true to it's name, I have a fairly good idea what it likely does, but I have no idea how it does it specifically.

Regardless, worst case scenario, it's likely doing nothing. Worst case it's perhaps the cause of your issue?

Good luck with it. If you do go the Squid route, please feel free to ask if you have any questions. I wouldn't suggest that I'm an expert, but I went for a fairly deep dive when I set up the squid proxy for TKLDev v17.x. So if you have questions soon, hopefully I'll still remember something! :)

Actually, just before I posted, I had a quick look at the squid-deb-proxy-client package page and apparently it scans for network services published as _apt_proxy._tcp and will configure apt to use the proxy that it finds. In theory it probably should work fine (assuming apt-cacher-ng provides a "_apt_proxy._tcp" network service), although it's probably just as easy to manually configure it yourself (unless you plan to have fallback servers, etc).

Hi Jeremy, thanks for the explanation. I'm actually a bit more tempted than you might think to eventually set up something like Squid as a package cache, apt-cacher-ng works okay but it's got a few bugs and seems a bit worse for wear.

I think for the time being I'm going to uninstall squid-deb-proxy-client and see if that magically brings things back into working order, then decide what to do based on the results of that.

Thanks for the additional info. I wasn't aware that squid-deb-proxy-client works that way (i.e. autoconfigured proxy). That's actually pretty cool and I might have a play with it at some point. TBH, I was wondering what the point of it was and that makes much more sense to me.

squid-deb-proxy-client actually works great when it works, but Avahi based discovery is not easy to debug when it decides to stop working, which it occasionally does for mysterious reasons.

I started using it long ago to avoid doing more manual configuration, but such a thing mostly makes sense for laptops (use the proxy if discovered, continue without it if not), not for machines that stay in one place all the time.

By default Squid is configured to cache HTTP and HTTPS traffic. As you note, it essentially performs a MITM (man in the middle). It terminates the "real" TLS connection with the external remote, then uses it's own CA (which you need to have preloaded into the system for other apps to trust the connection) to generate new certs on the fly.

FWIW, when I developed the updated caching system using squid for TKLDev v17.x, I post the squid config info on *Nix StackExchange. I can also point you towards the TKLDev code that leverages it if you wish.

Jeremy, I'm curious if you know offhand whether Squid can be a MITM caching proxy for downloading packages via HTTPS.

Currently I'm stuck downloading packages over HTTP so apt-cacher-ng can cache them, it'd be great if I could manage to stop downloading in cleartext as part of switching away from apt-cacher-ng.

No worries if you don't know, just figured I'd ask if you have experience (because if so you can warn me if it's a huge can of worms)

It is Debian specific although AFAIK, it should work the same on any Debian derived systems (including Ubuntu - although the package name may be different). If you are using an alternate distro, then I'm not sure, although worst case scenario, you could build it from source (with the relevant build option enabled).

Adding: that question is for my personal network, not in relation to use of HTTPS with TKLDev.