I have a Postgres server running, and at the end of September my Let's Encrypt certificate stopped working for clients on some platforms (mainly Apple). Chrome on Windows thinks it's fine, but if I use whynopadlock to check it, I get an error "Invalid Intermediate You have an invalid or missing intermediate (bundle) certificate. This may not break your padlock on all browsers, but will on others. Please contact your SSL Vendor for assistance with this error."

I've tried renewing my cert via dehydrated, and I've checked that all the components of the chain are present in the cert file (server cert, intermediate for Lets Encrypt, and root cert). For some reason, clients don't see the chain and they consider the certificate "not trusted".

What do I need to do to get the right cert from dehydrated? Thanks!

--C

Forum: 

So, I was able to finally find the magic combination of certificates and lighttpd settings to get it working, but it involved a lot more trial-and-error and downloading intermediate certificates manually than I expected.

The certificate I have contains the full chain (and the key of course), but for some reason only the leaf certificate gets served by lighttpd unless I add a separate intermediate cert file and point to it with the ssl.ca-cert parameter in lighttpd.conf. Isn't it supposed to be sufficient to have the full chain in the main .pem file? What am I missing? And how do I make sure things stay on track the next time dehydrated tries to update the certificate?

 

Jeremy Davis's picture

Hi Clay, sorry to hear of your TLS cert issues. I was under the impression that we include (and get from Dehydrated) the fullchain cert by default. It seems that either something has changed with Let's Encrypt or for some reason, Lighty isn't using it by default.

Firstly it would be useful to understand what you needed to do to get it working. Do you recall exactly? Or can you at least provide a rough overview of what was required to get to where you are?

Next up, it would be useful to understand what version your appliance is. If you are unsure, then please run 'turnkey-version'. Please also share any other details that may be relevant (e.g. any major OS changes that you may have made since initial install).

It would also be useful to understand that relevant software versions you have. So please run 'apt update' and post the output of:

apt policy confconsole dehydrated

It's also possible that there are relevant updates that haven't yet been installed (by default TurnKey only auto installs security updates). So to list the available updates, run (and post back the output of):

apt list --upgradeable

Also, I'm fairly sure it's not relevant, but I am aware that the Let's Encrypt root certificate has expired. But I'm pretty sure that Debian released updates to resolve that issue and it doesn't really match what you've reported anyway (but thought it might be worth noting).

Here's what I did to get it working:

  • Went to the Let's Encrypt web site and got a copy of the intermediate certificate (R3)
  • Put that on my server alongside the full chain cert I was using
    • Verified that cert.pem includes a copy of the contents of lets_encrypt_r3.pem, so it is the full chain, but for whatever reason wasn't working as such
  • In my ssl-params.conf file under lighttpd, added
    ssl.ca-file = "/etc/ssl/private/lets_encrypt_r3.pem"
  • restarted lighttpd

Command turnkey-version returns: 

turnkey-postgresql-15.0-stretch-amd64

No major OS updates that I'm aware of. As part of the troubleshooting process I tried updating confconsole, but it said I already had the latest version.

Command apt policy confconsole dehydrated returns:

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

confconsole:
  Installed: 1.1.2
  Candidate: 1.1.2
  Version table:
 *** 1.1.2 100
        100 /var/lib/dpkg/status
     1.1.0+2+g6c2aad9 999
        999 http://archive.turnkeylinux.org/debian stretch/main amd64 Packages
dehydrated:
  Installed: 0.6.2-2+deb10u1~deb9u1
  Candidate: 0.6.2-2+deb10u1~deb9u1
  Version table:
 *** 0.6.2-2+deb10u1~deb9u1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

Command apt list --upgradeable returns:

base-files/oldoldstable 9.9+deb9u13 amd64 [upgradable from: 9.9+deb9u4]
debian-archive-keyring/oldoldstable 2017.5+deb9u1 all [upgradable from: 2017.5]
dpkg/oldoldstable 1.18.25 amd64 [upgradable from: 1.18.24]
gettext-base/oldoldstable 0.19.8.1-2+deb9u1 amd64 [upgradable from: 0.19.8.1-2]
gnupg/oldoldstable 2.1.18-8~deb9u4 amd64 [upgradable from: 2.1.18-8~deb9u2]
gnupg-agent/oldoldstable 2.1.18-8~deb9u4 amd64 [upgradable from: 2.1.18-8~deb9u2]
gpgv/oldoldstable 2.1.18-8~deb9u4 amd64 [upgradable from: 2.1.18-8~deb9u2]
grub-common/oldoldstable 2.02~beta3-5+deb9u2 amd64 [upgradable from: 2.02~beta3-5]
grub-pc/oldoldstable 2.02~beta3-5+deb9u2 amd64 [upgradable from: 2.02~beta3-5]
grub-pc-bin/oldoldstable 2.02~beta3-5+deb9u2 amd64 [upgradable from: 2.02~beta3-5]
grub2-common/oldoldstable 2.02~beta3-5+deb9u2 amd64 [upgradable from: 2.02~beta3-5]
hdparm/oldoldstable 9.51+ds-1+deb9u1 amd64 [upgradable from: 9.51+ds-1]
libc-bin/oldoldstable 2.24-11+deb9u4 amd64 [upgradable from: 2.24-11+deb9u3]
libc-l10n/oldoldstable 2.24-11+deb9u4 all [upgradable from: 2.24-11+deb9u3]
libc6/oldoldstable 2.24-11+deb9u4 amd64 [upgradable from: 2.24-11+deb9u3]
libcups2/oldoldstable 2.2.1-8+deb9u6 amd64 [upgradable from: 2.2.1-8+deb9u2]
libdap23/oldoldstable 3.18.2-2+deb9u1 amd64 [upgradable from: 3.18.2-2]
libdapclient6v5/oldoldstable 3.18.2-2+deb9u1 amd64 [upgradable from: 3.18.2-2]
libdapserver7v5/oldoldstable 3.18.2-2+deb9u1 amd64 [upgradable from: 3.18.2-2]
libdbus-1-3/oldoldstable 1.10.32-0+deb9u1 amd64 [upgradable from: 1.10.28-0+deb9u1]
libfribidi0/oldoldstable 0.19.7-1+deb9u1 amd64 [upgradable from: 0.19.7-1+b1]
libfuse2/oldoldstable 2.9.7-1+deb9u2 amd64 [upgradable from: 2.9.7-1+deb9u1]
libglib2.0-0/oldoldstable 2.50.3-2+deb9u2 amd64 [upgradable from: 2.50.3-2]
libidn11/oldoldstable 1.33-1+deb9u1 amd64 [upgradable from: 1.33-1]
libmspack0/oldoldstable 0.5-1+deb9u3 amd64 [upgradable from: 0.5-1+deb9u2]
libperl5.24/oldoldstable 5.24.1-3+deb9u7 amd64 [upgradable from: 5.24.1-3+deb9u5]
libseccomp2/oldoldstable 2.3.1-2.1+deb9u1 amd64 [upgradable from: 2.3.1-2.1]
liburiparser1/oldoldstable 0.8.4-1+deb9u1 amd64 [upgradable from: 0.8.4-1]
libwayland-client0/oldoldstable 1.12.0-1+deb9u1 amd64 [upgradable from: 1.12.0-1]
libwayland-cursor0/oldoldstable 1.12.0-1+deb9u1 amd64 [upgradable from: 1.12.0-1]
libwayland-server0/oldoldstable 1.12.0-1+deb9u1 amd64 [upgradable from: 1.12.0-1]
libxcursor1/oldoldstable 1:1.1.14-1+deb9u2 amd64 [upgradable from: 1:1.1.14-1+deb9u1]
libxml-security-c17v5/oldoldstable 1.7.3-4+deb9u3 amd64 [upgradable from: 1.7.3-4+deb9u1]
libxslt1.1/oldoldstable 1.1.29-2.1+deb9u2 amd64 [upgradable from: 1.1.29-2.1]
locales/oldoldstable 2.24-11+deb9u4 all [upgradable from: 2.24-11+deb9u3]
monit/oldoldstable 1:5.20.0-6+deb9u1 amd64 [upgradable from: 1:5.20.0-6]
multiarch-support/oldoldstable 2.24-11+deb9u4 amd64 [upgradable from: 2.24-11+deb9u3]
open-vm-tools/oldoldstable 2:10.1.5-5055683-4+deb9u2 amd64 [upgradable from: 2:10.1.5-5055683-4+deb9u1]
open-vm-tools-dkms/oldoldstable 2:10.1.5-5055683-4+deb9u2 all [upgradable from: 2:10.1.5-5055683-4+deb9u1]
openssh-client/oldoldstable 1:7.4p1-10+deb9u7 amd64 [upgradable from: 1:7.4p1-10+deb9u6]
openssh-server/oldoldstable 1:7.4p1-10+deb9u7 amd64 [upgradable from: 1:7.4p1-10+deb9u6]
openssh-sftp-server/oldoldstable 1:7.4p1-10+deb9u7 amd64 [upgradable from: 1:7.4p1-10+deb9u6]
perl/oldoldstable 5.24.1-3+deb9u7 amd64 [upgradable from: 5.24.1-3+deb9u5]
perl-base/oldoldstable 5.24.1-3+deb9u7 amd64 [upgradable from: 5.24.1-3+deb9u5]
perl-modules-5.24/oldoldstable 5.24.1-3+deb9u7 all [upgradable from: 5.24.1-3+deb9u5]
postfix/oldoldstable 3.1.15-0+deb9u1 amd64 [upgradable from: 3.1.8-0+deb9u1]
postfix-sqlite/oldoldstable 3.1.15-0+deb9u1 amd64 [upgradable from: 3.1.8-0+deb9u1]
rsync/oldoldstable 3.1.2-1+deb9u2 amd64 [upgradable from: 3.1.2-1+deb9u1]
shared-mime-info/oldoldstable 1.8-1+deb9u1 amd64 [upgradable from: 1.8-1]
ssh/oldoldstable 1:7.4p1-10+deb9u7 all [upgradable from: 1:7.4p1-10+deb9u6]
tklbam/stretch 1.4.1+37+g8117cd6 all [upgradable from: 1.4.1+32+g07acc1c]
unzip/oldoldstable 6.0-21+deb9u2 amd64 [upgradable from: 6.0-21]

Thanks for taking the time to look into it!

--Clay

Jeremy Davis's picture

TBH, I'm still not at all clear why this has occurred now; out of the blue.

For background, the default TurnKey certificate file (/etc/ssl/private/cert.pem) actually includes the certificate, the (secret) key and the DH parameters. The certificate should be the full certificate chain which should definitely be enough. I may be wrong, but I think this issue you've hit may actually be related to the expiry of the root LE certificate?!

Regardless, looking at what you've done, your setup should continue to function fine. The fact that the root cert is in a separate file (that Confconsole/Dehydrated/Let's Encrypt won't touch) means that future Dehydrated/Let's Encrypt certificate updates should "just work". I would suggest testing that prior to the next expiry of your certificate though. That way you can be in front of the game!

To do that, try this:

/usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper --force

Thanks too for sharing "whynopadlock". I hadn't come across that one before and I'll add that to my testing regime. FWIW we've always just used SSL Labs.

Thanks Jeremy! It sounds like my understanding isn't too far off the mark. I still wonder why the fullchain cert is not enough any more, but as long as I have a working configuration, and the auto update shouldn't change the part I added to get it working, I'm happy.

badco's picture

FYI

https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

I was aware if this because my pfsense gateway was giving me notifications about the expiration.

Thanks. I did see that, and I assume that it's somehow related. But the cert I'm using with ssl.ca-file is the same one that's in the fullchain cert I'm using. That's the part that I still don't understand. But I have a configuration that works so I'm satisfied for now.

Add new comment