On a virtual machine with a private IPv4, and a public IPv6 interfaces/addresses, running the gitea appliance

root@g ~# turnkey-version
turnkey-gitea-17.1-bullseye-amd64
root@g ~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

Dehydrated fails to obtain a cert with a time out error.

Investigation shows that the python script that Dehydrated spawns is only listening on 0.0.0.0:80

Disabling the IPv4 interface (ifdown) and keeping the IPv6 as the only option for connectivity changes nothing.

The DNS is set up properly with the host name pointing to a single IPv6 address. The issue is that python is not listening on IPv6.

Here are the listening processes while waiting for the verification:

<!--break-->

root@g ~# ss -lptn
State              Recv-Q             Send-Q                           Local Address:Port                            Peer Address:Port             Process
LISTEN             0                  80                                   127.0.0.1:3306                                 0.0.0.0:*                 users:(("mariadbd",pid=481,fd=48))
LISTEN             0                  4096                                   0.0.0.0:5355                                 0.0.0.0:*                 users:(("systemd-resolve",pid=137,fd=12))
LISTEN             0                  5                                      0.0.0.0:80                                   0.0.0.0:*                 users:(("python3",pid=11630,fd=5))
LISTEN             0                  4096                                 127.0.0.1:10000                                0.0.0.0:*                 users:(("miniserv.pl",pid=9948,fd=5))
LISTEN             0                  4096                             127.0.0.53%lo:53                                   0.0.0.0:*                 users:(("systemd-resolve",pid=137,fd=17))
LISTEN             0                  128                                    0.0.0.0:22                                   0.0.0.0:*                 users:(("sshd",pid=369,fd=3))
LISTEN             0                  1                                    127.0.0.1:9977                                 0.0.0.0:*                 users:(("python3",pid=11630,fd=3))
LISTEN             0                  100                                  127.0.0.1:25                                   0.0.0.0:*                 users:(("master",pid=4686,fd=13))
LISTEN             0                  128                                  127.0.0.1:12319                                0.0.0.0:*                 users:(("shellinaboxd",pid=9132,fd=4))
LISTEN             0                  4096                                   0.0.0.0:12321                                0.0.0.0:*                 users:(("stunnel4",pid=9081,fd=9))
LISTEN             0                  4096                                      [::]:5355                                    [::]:*                 users:(("systemd-resolve",pid=137,fd=14))
LISTEN             0                  128                                       [::]:22                                      [::]:*                 users:(("sshd",pid=369,fd=4))
LISTEN             0                  4096                                         *:12320                                      *:*                 users:(("stunnel4",pid=9101,fd=9))

Dehydrated log:

root@g ~# /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper -r

[2022-12-08 21:57:02] dehydrated-wrapper: INFO: started
[2022-12-08 21:57:02] dehydrated-wrapper: INFO: No process found listening on port 80; continuing
[2022-12-08 21:57:02] dehydrated-wrapper: INFO: running dehydrated
ERROR: Challenge is invalid! (returned: invalid) (result: ["type"]      "http-01"
["status"]      "invalid"
["error","type"]        "urn:ietf:params:acme:error:connection"
["error","detail"]      "2a11:4c8:173:462::16:1: Fetching http://gt.domain.com/.well-known/acme-challenge/SgxwEdm4aWjV5eQtAfH_UEd9qtrGnL1XWtwU1pq4r: Timeout during connect (likely firewall problem)"
["error","status"]      400
["error"]       {"type":"urn:ietf:params:acme:error:connection","detail":"2a11:4c8:173:462::16:1: Fetching http://gt.domain.com/.well-known/acme-challenge/SgxwEdm4aWjV5eQtAfH_UEd9qtrGnL1XWtwU1pq4r: Timeout during connect (likely firewall problem)","status":400}
["url"] "https://acme-v02.api.letsencrypt.org/acme/chall-v3/185464525747/IN0IoQ"
["token"]       "SgxwEdm4aWjV5eQtAfH_UEd9qtrGnL1XWtwU1pq4r"
["validationRecord",0,"url"]    "http://gt.domain.com/.well-known/acme-challenge/SgxwEdm4aWjV5eQtAfH_UEd9qtrGnL1XWtwU1pq4r"
["validationRecord",0,"hostname"]       "gt.domain.com"
["validationRecord",0,"port"]   "80"
["validationRecord",0,"addressesResolved",0]    "2a11:4c8:173:462::16:1"
["validationRecord",0,"addressesResolved"]      ["2a11:4c8:173:462::16:1"]
["validationRecord",0,"addressUsed"]    "2a11:4c8:173:462::16:1"
["validationRecord",0]  {"url":"http://gt.domain.com/.well-known/acme-challenge/SgxwEdm4aWjV5eQtAfH_UEd9qtrGnL1XWtwU1pq4r","hostname":"gt.domain.com","port":"80","addressesResolved":["2a11:4c8:173:462::16:1"],"addressUsed":"2a11:4c8:173:462::16:1"}
["validationRecord"]    [{"url":"http://gt.domain.com/.well-known/acme-challenge/SgxwEdm4aWjV5eQtAfH_UEd9qtrGnL1XWtwU1pq4r","hostname":"gt.domain.com","port":"80","addressesResolved":["2a11:4c8:173:462::16:1"],"addressUsed":"2a11:4c8:173:462::16:1"}]
["validated"]   "2022-12-08T21:57:07Z")
[2022-12-08 21:57:19] dehydrated-wrapper: FATAL: dehydrated exited with a non-zero exit code.
[2022-12-08 21:57:19] dehydrated-wrapper: WARNING: Python is still listening on port 80
[2022-12-08 21:57:19] dehydrated-wrapper: INFO: attempting to kill add-water server
[2022-12-08 21:57:19] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert, key and combined files.
[2022-12-08 21:57:19] dehydrated-wrapper: INFO: (Re)starting stunnel4@webmin.service
[2022-12-08 21:57:19] dehydrated-wrapper: INFO: (Re)starting stunnel4@shellinabox.service
[2022-12-08 21:57:19] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.

Any idea how to get this to work?

Thank you.

Note: names and addresses changed for privacy.

Forum: 
Jeremy Davis's picture

It's an oversight on our behalf, but you're right, it certainly should support it, ideally OOTB - but at least via a config option. Let me have a play and I'll get back to you.

Jeremy Davis's picture

It's a pretty easy fix. I didn't actually test getting a certificate, but I confirmed that it listens on IPv6 (and still on IPv4 too) after this change. Please be aware that if you retried lots of times, there is a chance that your IP has been blacklisted. Unfortunately, if that's happened, you'll either need to wait - or get a new IP :(

So to apply the workaround, you'll need to edit /usr/lib/confconsole/plugins.d/Lets_Encrypt/add-water-srv. On the very last line, change what is there:

run(host='0.0.0.0', port=80)

to

run(host='::', port=80)

And you should be good to go! :)

I've opened an issue on our tracker so it doesn't get forgotten and I've also opened a PR that will fix it in the code base.

Thank you so much for the fast responses, Jeremy.

Much appreciated!

Jeremy Davis's picture

I was fairly confident that it would work, but thanks for the confirmation.

Also thanks too for the quality of your bug report. With a detailed report like that with lots of info, you really do make my life easier. If only everyone where so good at sharing info about problems they hit! :)

Add new comment