TK - Lets-Encrypt - dehydrated-wrapper - null $CTYPE

It gives the full info when it works, but not when it doesn't!? (as it should).

TBH, I don't know why that's the case but there is a line in your logs that gives me some idea (albeit not real clarity on the issue). This is the line I'm talking about:

[2024-06-23 10:14:02] dehydrated-wrapper: FATAL: Unexpected challenge type:

Note the colon at the end. That's where the full dehydrated error log should appear. Obviously it's not...

As it's only failing when it's run from the cron job, I suspect that it's something related to the cron job itself. FYI cron jobs run in a limited environment so it seems likely that there is something missing from the cron env.

Out of interest, does the Let's Encrypt error occur within 24 hours of manually running it? I ask because beyond the cron env, there is another fundamental difference between running it manually and via cron.

When it's run manually, it will force update of the cert - regardless of when the current cert will expire. Via cron it should only attempt to update the cert if it expires in 30 days or less. New certs are valid for 90 days. Given that log line I quote above, if the cron failure happens within 24 hours of the manual update, then that's another unexpected and weird component of this problem.

TBH, it's a real mystery why no one else has reported the issue you're hitting. FWIW we're dogfooding our LE integration - this server uses the same Let's Encrypt process that you are. Although actually, perhaps your server is a different TurnKey version and/or has a different version of Confconsole? IIRC this site was actually built before I built the publicly released images.

Regardless, can you please confirm the TurnKey version of your server and the version of Confconsole it's running. Here's how to do that respectively:

turnkey-version
apt list confconsole

Re your suspicion that it's a $CTYPE issue, why do you think that? FYI if it's anything to do with the below message when installing/updating packages, then that's not an issue:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = "en_US.UTF-8",
    LC_ALL = (unset),
    LC_TIME = "en_AU.utf8",
    LC_MONETARY = "en_AU.utf8",
    LC_CTYPE = "C",
    LC_COLLATE = "C",
    LC_MEASUREMENT = "en_AU.utf8",
    LC_NUMERIC = "en_AU.utf8",
    LC_PAPER = "en_AU.utf8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = "en_US.UTF-8",
    LC_ALL = (unset),
    LC_TIME = "en_AU.utf8",
    LC_CTYPE = "C",
    LC_MONETARY = "en_AU.utf8",
    LC_COLLATE = "C",
    LC_MEASUREMENT = "en_AU.utf8",
    LC_NUMERIC = "en_AU.utf8",
    LC_PAPER = "en_AU.utf8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").

FYI, I copied that from a local v18.0 LAMP server I have running. I have not changed any of those server side. (Some are propagating via SSH from my local PC,

My apologies for misunderstanding the context of your comments. It never occurred to me that you were referring to the source code and I didn't recall the 'CTYPE' variable name off the top of my head. Thanks for the clarification, as well as the extra info. Alos apologies that my previous response was so wildly off the mark.

Anyway, you are totally on the money! I've just run a few tests and can (easily) reproduce it. TBH, I'm not sure how that slipped through, it's a pretty serious issue. On face value it appears that it was a regression that I introduced when I fixed the issues we had with the DNS-01 challenge early in the v18.0 release.

I'll develop a fix and get back to you ASAP.

Thanks again, particularly for your persistence!

Thank you for your reply,

I was analysing the source code here and i found out that the message that you stated only appears in 3 cases. Either is http-01, dns-01 or neither as in lines 262 to 271:

case $CTYPE in
    http-01) cp "$SH_HOOK_HTTP" "$CC_HOOK"
             PORT=80
             sed -i '\|^CHALLENGETYPE=|s|=.*|="http-01"|' "$CONFIG";;
    dns-01)  cp "$SH_HOOK_DNS" "$CC_HOOK"
             PORT=443
             sed -i '\|^CHALLENGETYPE=|s|=.*|="dns-01"|' "$CONFIG";;
    *)       fatal "Unexpected challenge type: $CTYPE"
esac
export CHALLENGETYPE=$CTYPE

Therefore my suspicious of $CTYPE == null. People tend to not report it for lazyness and also because they might not use the resource as often. As I have seen in the foruns, they tend to use dns-01 more often than te basic http-01.

For your question about the cronjob: Yes it's daily at 06:08 am and this behavior persists when triggering manually using:

/usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper --force

I already post it before but here it's:

$turnkey-version
turnkey-nextcloud-18.0-bookworm-amd64
$apt list confconsole
Listing... Done
confconsole/bookworm,now 2.1.5 all [installed]

Basically i'm just using the nextcloud ISO provided for proxmox. I tried to not tweek as much as possible with this ISO to maintain stability so I'm (almost) sure that no basics distro configurations where not removed or changed (apart for ssh.service which was completly removed and doesn't concern this problem).

Actually scratch that. Looks like my initial testing was poor... :( It seems this new version may have introduced a new regression **sigh**

I'll be back soon with another new build.

# download
wget /files/tmp/confconsole_2.1.5+2+g40d7d22_all.deb

# for extra security, check the sha512sum against what is posted below
sha512sum confconsole_2.1.5+2+g40d7d22_all.deb

# install
apt install ./confconsole_2.1.5+2+g40d7d22_all.deb

e5aa2a31aa6e9d60e03a26aed05e260f1969294a7f1f819a81a89037c33318b01f87dc09a869b02887dbd313bf5fa8c99cd96a52883c865a5be8ca820f34423e

Yay, thanks for confirming that the updated script works. Sorry about the broken link, not sure quite what happened there, but it was getting late.

Thanks again for reporting. Given my testing and your confirmation, I'll aim to rebuild and push the new build to our apt repo today.

I aim to post back and if/when you have time it'd be great if you could confirm the apt package works too.

Regardless, take care and I look forward to next time... :)

I took your commit from github and edited in my server.

Everything seems fine:

[2024-06-28 00:07:13] dehydrated-wrapper: INFO: started
[2024-06-28 00:07:13] dehydrated-wrapper: INFO: found apache2 listening on port 80
[2024-06-28 00:07:13] dehydrated-wrapper: INFO: stopping apache2
[2024-06-28 00:07:13] dehydrated-wrapper: INFO: running dehydrated
[2024-06-28 00:07:24] confconsole.hook.sh: SUCCESS: Cert request successful. Writing relevant files for mydomainexample.com.
[2024-06-28 00:07:25] confconsole.hook.sh: SUCCESS: Files written/created for mydomainexample.com: /usr/local/share/ca-certificates/cert.crt - /etc/ssl/private/cert.key - /etc/ssl/private/cert.pem.
[2024-06-28 00:07:25] dehydrated-wrapper: INFO: dehydrated complete
[2024-06-28 00:07:25] dehydrated-wrapper: WARNING: Python is still listening on port 80
[2024-06-28 00:07:25] dehydrated-wrapper: INFO: attempting to kill add-water server
[2024-06-28 00:07:25] dehydrated-wrapper: INFO: Cleaning backup cert & key
[2024-06-28 00:07:25] dehydrated-wrapper: INFO: (Re)starting apache2
[2024-06-28 00:07:25] dehydrated-wrapper: INFO: (Re)starting webmin.service
[2024-06-28 00:07:28] dehydrated-wrapper: INFO: dehydrated-wrapper completed successfully.

Thank you for your fast response.

I tried to download but got 404:

    wget /files/tmp/confconsole_2.1.5+5+g515d28d_all.deb
    /files/tmp/confconsole_2.1.5+5+g515d28d_all.deb
    Resolving www.turnkeylinux.org (www.turnkeylinux.org)... 2a06:98c1:3120::c, 2a06:98c1:3121::c, 188.114.97.12, ...
    Connecting to www.turnkeylinux.org (www.turnkeylinux.org)|2a06:98c1:3120::c|:443... connected.
    HTTP request sent, awaiting response... 404 Not Found
    2024-06-27 23:31:41 ERROR 404: Not Found.

That'll teach me for getting ahead of myself... Anyway, I've tested this better. So please try this one:

# download
wget /files/tmp/confconsole_2.1.5+5+g515d28d_all.deb

# for extra security, check the sha512sum against what is posted below
sha512sum confconsole_2.1.5+5+g515d28d_all.deb

# install
apt install ./confconsole_2.1.5+5+g515d28d_all.deb

SHA sum:

70cedcb717ff3eb872ce2e3d6b9c8f56fe4ef7f09abe32e5933fa2ebf4d9631f2058b8059f93df84ea9d4f81261078e1ca6e212cb934883b0aba656f01d69c4e

Look forward to hearing how your go.

FYI I finalised the new version of Confconsole (v2.1.6) last month and pushed to the TurnKey v18.x (Debian 12/Bookworm) repo.