Hi experts.

Trying to generate a certificate.  Used confconsole tool to get new certificate.  I get the following error, but unclear what exactly happened...

[2020-08-28 13:53:32] dehydrated-wrapper: FATAL: dehydrated exited with a non-zero exit code.

Unfortunately I have no real idea where to look for a log file that might guide me as to the nature of the error.

Any guidance where I should look would be gratefully appreciated.  In the meantime I am using a self-signed certificate, but would prefer of course to use a "real" certificate.

Thanks!

Forum: 

I looked at the /var/log/confconsole log, and it shows the following error:

 

File "/usr/lib/confconsole/plugins.d/Lets_Encrypt/add-water-client", line 41, in <module> soc.connect((host, port))

ConnectionRefusedError: [Errno 111] Connection refused

Here is my situation:

- I have everything on the VM listening on default ports

- I am using a DDNS through freedns.org, and am using redirection from my router to this VM, so that my external clients use https://xx.yy.zz:8888 to get to my WP. The router redirects to https://192.168.0.22

- /etc/resolv.conf is fine, since I can ping my DDNS name from a terminal (and in any case I can successfully use the web interface through my self-signed cert).

Is there any special reconfiguration of dehydrated required that I am not doing?

Appreciate your guidance.

Did some further digging.  I was using Pretty Simple SSL, which I deactivated.  Ran again, error.  I decided to use a REST client and manually entered the request.  The response I get indicates a firewall issue ("Timeout during connect (likely firewall problem)".  I'm not quite sure where that possible firewall issue might be

- my router

- Linux VM

- Windows host firewall

I want to eliminate TurnkeyLinux firewall possibilities first, and I can then look at my router and my Windows host.

Appreciate any guidance.

Jeremy Davis's picture

Your previous posts were in reference to our WordPress appliance and I just wanted to double check that in the case here too? Assuming that it is (or is at least one of our LAMP based appliances) it'd also be good to know what (if any) significant changes you may have made to the Apache config. If it's not WordPress (or LAMP) then the same question but related to the default web server.

Regardless, from what you've posted, it sounds like the add-water client (part of our super simple HTTP server for serving the Let's Encrypt challenges) is unable to connect to the add-water server.

To assist to troubleshoot this, could you please post the full output of the log file (IIRC it should be /var/log/confconsole/letsencrypt.log). Plus it might also be useful to check both the add-water and Apache status, and finally, which ports are being used and by what. So:

cat /var/log/confconsole/letsencrypt.log
systemctl status add-water
systemctl status apache2
netstat -tlnp

Hi Jeremy:

Many thanks for the quick response.  My apologies - I should have made clearer what appliance I was using.  Yes, I am using WordPress.  At this point, I have made no changes to the Apache config as I wanted to work with the OOTB appliance as much as possible.

I have several WP plugins installed as one might expect - Absolutely Glamorous Custom Admin, Akismet AntiSpam, New User Approve, Page Restrict, Really Simple SSL, Search & Replace, Simple Local Avatars, String Locator, Updraft Plus.  Based on some digging around, I have played around with activating/deactivating Really Simple SSL (when active, I get a "connection refused" error; when deactivated, I get the "timeout during connect" error described in tis thread).

The results you requested:

 

***********************************************************************
letsencrypt.log
***********************************************************************

[2020-08-27 23:38:11] dehydrated-wrapper: FATAL: dehydrated exited with a non-zero exit code.
[2020-08-27 23:38:11] dehydrated-wrapper: WARNING: Python is still listening on port 80
[2020-08-27 23:38:11] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert, key and combined files.
[2020-08-27 23:38:12] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.
Traceback (most recent call last):
  File "/usr/lib/confconsole/plugins.d/Lets_Encrypt/add-water-client", line 41, in <module>
    sock.connect((host, port))
ConnectionRefusedError: [Errno 111] Connection refused
[2020-08-27 23:38:46] dehydrated-wrapper: FATAL: dehydrated exited with a non-zero exit code.
[2020-08-27 23:38:46] dehydrated-wrapper: WARNING: Python is still listening on port 80
[2020-08-27 23:38:46] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert, key and combined files.
[2020-08-27 23:38:46] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.
ERROR: Challenge is invalid! (returned: invalid) (result: {
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:connection",
    "detail": "Fetching http://lanew.privatedns.org/.well-known/acme-challenge/GHgXGtJead5s_dO1P... Timeout during connect (likely firewall problem)",
    "status": 400
  },
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/6816056597/hMPo9A",
  "token": "GHgXGtJead5s_dO1P1vq8wxc0SsJTAph-2SVkIPbydU",
  "validationRecord": [
    {
      "url": "http://lanew.privatedns.org/.well-known/acme-challenge/GHgXGtJead5s_dO1P...,
      "hostname": "lanew.privatedns.org",
      "port": "80",
      "addressesResolved": [
        "198.133.9.12"
      ],
      "addressUsed": "198.133.9.12"
    }
  ]
})
[2020-08-28 13:53:32] dehydrated-wrapper: FATAL: dehydrated exited with a non-zero exit code.
[2020-08-28 13:53:32] dehydrated-wrapper: WARNING: Python is still listening on port 80
[2020-08-28 13:53:32] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert, key and combined files.
[2020-08-28 13:53:32] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.
ERROR: Challenge is invalid! (returned: invalid) (result: {
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:connection",
    "detail": "Fetching http://lanew.privatedns.org/.well-known/acme-challenge/gzO2UyRJ7aM7e7pFq... Timeout during connect (likely firewall problem)",
    "status": 400
  },
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/6832537209/xfkZLQ",
  "token": "gzO2UyRJ7aM7e7pFqYgm7Fb1l3e0uqdauqljPINsmKI",
  "validationRecord": [
    {
      "url": "http://lanew.privatedns.org/.well-known/acme-challenge/gzO2UyRJ7aM7e7pFq...,
      "hostname": "lanew.privatedns.org",
      "port": "80",
      "addressesResolved": [
        "198.133.9.12"
      ],
      "addressUsed": "198.133.9.12"
    }
  ]
})
[2020-08-28 18:00:00] dehydrated-wrapper: FATAL: dehydrated exited with a non-zero exit code.
[2020-08-28 18:00:00] dehydrated-wrapper: WARNING: Python is still listening on port 80
[2020-08-28 18:00:00] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert, key and combined files.
[2020-08-28 18:00:00] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.

***********************************************************************
add-water status
***********************************************************************

* add-water.service - Add Water
   Loaded: loaded (/lib/systemd/system/add-water.service; static; vendor preset: enabled)
   Active: inactive (dead)

Aug 28 13:53:16 wordpress systemd[1]: Started Add Water.
Aug 28 13:53:32 wordpress systemd[1]: Stopping Add Water...
Aug 28 13:53:32 wordpress systemd[1]: add-water.service: Main process exited, code=killed, status=15/TERM
Aug 28 13:53:32 wordpress systemd[1]: add-water.service: Succeeded.
Aug 28 13:53:32 wordpress systemd[1]: Stopped Add Water.
Aug 28 17:59:44 wordpress systemd[1]: Started Add Water.
Aug 28 18:00:00 wordpress systemd[1]: Stopping Add Water...
Aug 28 18:00:00 wordpress systemd[1]: add-water.service: Main process exited, code=killed, status=15/TERM
Aug 28 18:00:00 wordpress systemd[1]: add-water.service: Succeeded.
Aug 28 18:00:00 wordpress systemd[1]: Stopped Add Water.

***********************************************************************
apache2 status
***********************************************************************

* apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-08-28 18:00:00 UTC; 6h ago
     Docs: https://httpd.apache.org/docs/2.4/
  Process: 12555 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
  Process: 13914 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
 Main PID: 12560 (apache2)
    Tasks: 6 (limit: 1143)
   Memory: 50.8M
   CGroup: /system.slice/apache2.service
           |-12560 /usr/sbin/apache2 -k start
           |-13920 /usr/sbin/apache2 -k start
           |-13921 /usr/sbin/apache2 -k start
           |-13922 /usr/sbin/apache2 -k start
           |-13923 /usr/sbin/apache2 -k start
           `-13924 /usr/sbin/apache2 -k start

Aug 28 18:00:00 wordpress systemd[1]: Starting The Apache HTTP Server...
Aug 28 18:00:00 wordpress systemd[1]: Started The Apache HTTP Server.
Aug 29 00:00:01 wordpress systemd[1]: Reloading The Apache HTTP Server.
Aug 29 00:00:01 wordpress systemd[1]: Reloaded The Apache HTTP Server.


*******************************************************************************
netstat -tlnp
*******************************************************************************

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      652/mysqld
tcp        0      0 127.0.0.1:10000         0.0.0.0:*               LISTEN      12643/perl
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      482/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      899/master
tcp        0      0 127.0.0.1:12319         0.0.0.0:*               LISTEN      12619/shellinaboxd
tcp        0      0 0.0.0.0:12320           0.0.0.0:*               LISTEN      12581/stunnel4
tcp        0      0 0.0.0.0:12321           0.0.0.0:*               LISTEN      12601/stunnel4
tcp6       0      0 :::80                   :::*                    LISTEN      12560/apache2
tcp6       0      0 :::22                   :::*                    LISTEN      482/sshd
tcp6       0      0 :::443                  :::*                    LISTEN      12560/apache2
tcp6       0      0 :::12322                :::*                    LISTEN      12560/apache2

 

Jeremy Davis's picture

Thanks for the additional info.

TBH, the actual issue seems unclear to me... :( Bottom line is that the Let's Encrypt servers aren't able to confirm that the challenge is being served.

Your letsencrypt.log suggests that there is (or at least was) a condition occurring which shouldn't be, but the actual issue is unclear. The stacktrace that appears in the log suggests that the add-water client is unable to connect to the add-water server. I.e.:

Traceback (most recent call last):
  File "/usr/lib/confconsole/plugins.d/Lets_Encrypt/add-water-client", line 41, in 
    sock.connect((host, port))
ConnectionRefusedError: [Errno 111] Connection refused

But I note that is an older entry, and only appears once. So perhaps that's a false concern?! Unfortunately the info from the add-water service at that time is not displayed, so I'm not sure what actually happened there.

TBH, I think that this may be a red herring... I'm post more below.

BTW, I just wanted to reiterate that I tried using the REST API directly on my Windows host machine as follows (using the Firefox REST Client):

GET: https://acme-v02.api.letsencrypt.org/acme/chall-v3/6832537209/xfkZLQ

Request: {
"validationRecord": [
{
"url": "http://lanew.privatedns.org/.well-known/acme-challenge/gzO2UyRJ7aM7e7pFq...,
"hostname": "lanew.privatedns.org",
"port": "80",
"addressesResolved": [
"198.133.9.12"
],
"addressUsed": "198.133.9.12"
}
]
}

Response: 

{
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:connection",
    "detail": "Fetching http://lanew.privatedns.org/.well-known/acme-challenge/gzO2UyRJ7aM7e7pFq... Timeout during connect (likely firewall problem)",
    "status": 400
  },
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/6832537209/xfkZLQ",
  "token": "gzO2UyRJ7aM7e7pFqYgm7Fb1l3e0uqdauqljPINsmKI",
  "validationRecord": [
    {
      "url": "http://lanew.privatedns.org/.well-known/acme-challenge/gzO2UyRJ7aM7e7pFq...,
      "hostname": "lanew.privatedns.org",
      "port": "80",
      "addressesResolved": [
        "198.133.9.12"
      ],
      "addressUsed": "198.133.9.12"
    }
  ]
}

 

 

Just for the hell of it, I tried POST...  The response I got was:

{
  "type": "urn:ietf:params:acme:error:malformed",
  "detail": "Parse error reading JWS",
  "status": 400
}

 

Not sure what method is actually being used - I had just assumed GET.  For POST I'm trying to understand what is malformed.

Jeremy Davis's picture

I checked your domain and as per what you've posted above, it points to the IP listed here. But It appears that either your server is no longer running, or perhaps you have a dynamic IP address and the DNS hasn't been updated?

So seeing as I can't access your server via the public IP that the DNS record points to, perhaps that's the issue? Or perhaps there is something blocking port 80 externally? (E.g. some ISPs block common internet ports or perhaps you're router has an auto-enabled firewall?).

So the next thing to double check is that the DNS is correct and points to your public IP address. And also check whether port 80 is open externally.

Essentially how it works is that the Let's Encrypt client (we use Dehydrated) connects to Let's Encrypt and asks for a token. Your server then needs to serve the token is the via a predefined URL. I.e. in this case it's http://lanew.privatedns.org/.well-known/acme-challenge/gzO2UyRJ7aM7e7pFqYgm7Fb1l3e0uqdauqljPINsmKI: The Let's Encrypt server tries to connect to that URL and confirm that it's serving the right token. If it is, it will then send a certificate. If it's not then it will error out...

If you look at the Let's Encrypt response:

"detail": "Fetching http://lanew.privatedns.org/.well-known/acme-challenge/gzO2UyRJ7aM7e7pFq... Timeout during connect (likely firewall problem)",
"status": 400

That essentially notes that it can't connect to your server via the specific URL. To reiterate my above points, the cause of that may be:

  • DNS not pointing to the correct IP address.
  • Port 80 of your public IP being blocked for some reason.
  • The correct challenge not being served by your server.

Other than the one time it failed (noted in the log), everything else suggests that add-water is working as expected (although it's not 100% clear that it hosted the correct challenge, we probably should tweak it so that we can double check that). So I think that the first 2 possibilities are worth further investigation.

Re the results you got from Firefox (on Windows), the GET result is expected (it gave a URL to serve, but then couldn't connect to it). As for the POST result, I'm not 100% sure, but AFAIK JWT tokens are generally used to protect against XSS attacks.

It's also well worth being aware that there is rate limiting on the main Let's Encrypt API end point (CA="https://acme-v02.api.letsencrypt.org/directory") . If you try and fail too many times then your domain will be temporarily blacklisted! For testing, I encourage you to try via the "staging" end point. You can set that by adding this line to the config file (/etc/dehydrated/confconsole.config):

CA="https://acme-staging-v02.api.letsencrypt.org/directory"

Whilst it "just works" for most people, it might be a good idea if we do a "test run" with the staging end point first by default, then only do it against the "proper" end point once that is successful...

Hi Jeremy:

Thanks for the update.

I am comfortable that the server is up and running - I can use incognito mode and serve up the default WP landing page using https://lanew.privatedns.org:8002 (my router takes care of the port translation and forwarding requests to the TurnkeyLinux VM).

I am also fine with DNS since the IP address registered with the DDNS service matches the one returned by whatsmyip.

So the only thing that could be causing an issue wouldbe the port translation? I do not wish to use default ports which is why I chose to use port 8002. My initial thoughts were to leave the TurnkeyLinux configuration unchanged, and use NAT to make the port change.

I tried "telnet lanew.privatedns.org 80" from my Windows host, but as I expected that does not work, since my router looks to route to port 8002.  However (also as I would expect), "telnet 192.168.0.22 80" works.

So that suggests to me that using lanew.privatedns.org in the url to get the challenge token will be a problem.

Is it feasible to change the default port used to be 8002 instead of 80 directly in TurnkeyLinux?

So far, the only issue I've encountered has been with letsencrypt.

Jeremy Davis's picture

Let's Encrypt requires the challenge be served via port 80. AFAIK you can workaround that and serve via a non standard port, but it still requires that Let's Encrypt has access to port 80 (and it redirects to the non-standard port you wish to serve the challenge on).

The other option for free Let's Encrypt is the DNS challenge. But that requires that you have direct access to the domain DNS records, so will only work with a domain that you own. I'm not super familiar with it, nor is TurnKey pre-configured to leverage it, but it is possible (when you have direct access to creating of new DNS TXT records).

To explicitly answer your question, yes you could easily change the TurnKey server to use an alternate port (e.g. port 8002 instead of 80), but that still won't solve this issue as Let's Encrypt will only check the challenges by trying to connect to http://lanew.privatedns.org (i.e. port 80). So if you aren't serving the challenges via port 80, it will fail.

Thanks, Jeremy.

Yeah, I'll need to figure out something to allow me to use port80 on the VM.  My problem is that the router admin interface is on port 80.  So any NAT rule on my router will cause a problem, so I will have to think about this some more to figure out a clean way to do this.

I will keep you posted.

Regards...

 

SUCCESS!

I added a new firewall rule to my router's port forwarding to route inbound requests over port 80 to TurnkeyLinux, and retried confconsole's Let'sEncrypt command. And this generated the cert I needed.

I verified the cert by using a browser from a new incognito window and I see the proper Let's Encrypt cert.

Thanks for all your guidance!

 

Jeremy Davis's picture

Glad you got it working! :)

Add new comment