Ortho's picture
Ortho - Tue, 2019/10/01 - 16:11

Hello,

So I powered a Magento2 Turnkey VM back up after a few months of it being offline, and new I'd need to renew its certificate.

I ran the following:

/usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper

and got:

--------------

[2019-10-01 09:07:43] dehydrated-wrapper: INFO: started
[2019-10-01 09:07:44] dehydrated-wrapper: INFO: found apache2 listening on port 80
[2019-10-01 09:07:44] dehydrated-wrapper: INFO: stopping apache2
[2019-10-01 09:07:44] dehydrated-wrapper: INFO: running dehydrated
  + ERROR: An error occurred while sending post-request to https://acme-v01.api.letsencrypt.org/acme/new-authz (Status 400)

Details:
{
  "type": "urn:acme:error:badNonce",
  "detail": "JWS has no anti-replay nonce",
  "status": 400
}

[2019-10-01 09:07:46] dehydrated-wrapper: FATAL: dehydrated exited with a non-zero exit code.
[2019-10-01 09:07:46] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert & key.
[2019-10-01 09:07:46] dehydrated-wrapper: INFO: starting apache2
[2019-10-01 09:07:46] dehydrated-wrapper: INFO: starting stunnel4
[2019-10-01 09:07:47] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.
------------------

From what i can find while googling, it seems as though the dehydrated script isn't handling the nonce sequence properly.

Any idea on how to fix this?

Forum: 
Support
Tags: 
lets encrypt
magento2
Jeremy Davis's picture

Jeremy Davis - Wed, 2019/10/02 - 00:44

Yes, it appears that Let's Encrypt have changed their server config and the change has broken the older version of Dehydrated that we have been using (installed from Debian repos).

There is some discussion regarding this (and some other issues) in another thread (specifically this post and this one). But probably your best bet is to have a read of the full info in the relevant issue on GitHub.

FWIW, I have just updated the issue with a link to the relevant Debian bug and a note that it appears to be possible to just edit the Dehydrated script itself to resolve the issue. I haven't tested that myself, but I figured it was worth noting.

Regardless, I should probably do a blog post (and send out a newsletter) about this as it will hit all users eventually (as their certificates expire). I'll be speaking with Alon tonight (my time) and we'll decide on a plan of attack...

Ortho's picture

Ortho - Wed, 2019/10/02 - 15:24

I tried both changes, and updating dehydrated did seem to work after a reboot.

Unfortunately I've seemed to have run into an LE Rate limit (I hate how it doesn't tell you which limit you've hit).

I tried then to change the dehydrated url to the test url, but now I always just get 400 error.

Is there a way to run the dehydrated wrapper but in test mode just to validate that everything is working once the rate limit clears?

Jeremy Davis's picture

Jeremy Davis - Wed, 2019/10/02 - 23:53

AFAIK the Let's Encrypt rate limit is 20 per week.

It's not well documented, but Confconsole actually ships with a alternate config designed to be used against the "Let's Encrypt staging server". It was created with the intention of being "dropped in" instead of the default config (it overrides the default URL that it hits, so just adding the relevant lines in the config file is another option). You can find a note about it within a discussion on GitHub.

Having said that, it was a fair while ago since it was last tested (at least by me). So I'm not even 100% sure that it still works. It'd be great if you wanted to test it out and let us know...

It might be best to save the current config, just in case. I.e.:

mv /etc/dehydrated/confconsole.config /etc/dehydrated/confconsole.config.orig
cp cp /usr/share/confconsole/letsencrypt/dehydrated-staging-confconsole.config /etc/dehydrated/confconsole.config

Then give it a run:

/usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper

Good luck mate.

Igor's picture

Igor - Mon, 2019/10/07 - 17:29 
root@zosma ~# /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper
[2019-10-07 17:25:51] dehydrated-wrapper: INFO:INFO started
[2019-10-07 17:25:52] dehydrated-wrapper: INFO: found apache2 listening on port 80
[2019-10-07 17:25:52] dehydrated-wrapperpper: INFO: stopping apache2

[2019-10-07 17:25:53] dehydrated-wrapperpperapper: INFO: running dehydrated
  + ERROR: An error occurred while sending post-request to https://acme-staging.api.letsencrypt.org/acme/new-reg (Status 403)
    Details: {
        "type": "uot;urn:acme:error:unauthorized",
	"detail": "Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.",
        "status": 403
             }
[2019-10-07 17:25:56] dehydrated-wrapper: FATAL: dehydrated exited with a non-zero exit code.
[2019-10-07 17:25:56] dehydrated-wrapper: WARNING: Something went went wrong, restoring original cert & key.
[2019-10-07 17:25:56] dehydratedehydrated-wrapper: INFO: starting apache2
[2019-10-07 17:25:56] dehydrated-wrapper: INFO: starting stunnel4
[2019-10-07 17:25:57] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.
Jeremy Davis's picture

Jeremy Davis - Tue, 2019/10/08 - 00:52

Ah yes, that is a new error. Reading the error message suggests that we'll need to update the API URL that Dehydrated uses. According to a Let's Encrypt announcement disallowing new registrations for APIv1 wasn't supposed to happen until next month. Existing users have until June 2020 apparently, although this early breakage of v1 suggests that it's probably best not to wait...

I documented on the bug report how to install from upstream (it's a bit dirty, but in this case it's not "unsafe"). Not sure if you've done that, or the just "fixed" the single line (i.e. adjusting the single line in Dehydrated)?

Regardless, either of those will resolve the Nonce issue, but without further changes, it will still attempt to use the v1 API. I haven't done any testing yet, but a quick google suggests that we'll need to adjust the URL that Dehydrated connects to (stored in the CA variable). Currently it is:

CA="https://acme-v01.api.letsencrypt.org/directory"

For new users, that now needs to be:

CA="https://acme-v02.api.letsencrypt.org/directory"

It's probably also worth existing users changing that too and double checking that everything works as it should.

It's probably worth noting that there is now a newer version of Dehydrated in stretch-backports now (actually, it's the latest version). So installing from backports is another (cleaner) way to upgrade it.

As noted above, I'm yet to do any testing to confirm my understanding, but essentially, these steps should resolve the issue(s):

  1. upgrade dehydrated (either from upstream or via stretch-backports)
  2. update the confconsole hook script (from TurnKey's GH)
  3. update the API URL (to v2)

And then everything should be all systems go!

I'll aim to look into this further ASAP (within the next few days) but I'm currently a bit bogged down with other stuff.

PS - I hope you don't mind me editing your post to make it a bit easier to read... :)

Igor's picture

Igor - Tue, 2019/10/08 - 10:39
  1. Added stretch-backports repository to APT sources list.
  2. Updated from there packages certbot and dehydrated (and all related packages too).
  3. Replaced links in /etc/dehydrated / conf console.config:
CA="https://acme-v02.api.letsencrypt.org/directory"
CA_TERMS="https://acme-v02.api.letsencrypt.org/terms"
  1. Ran commands: 
/usr/bin/dehydrated --register --accept-terms
/usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper
Jeremy Davis's picture

Jeremy Davis - Wed, 2019/10/09 - 05:35

Thanks for the neat and tidy roundup Igor, that's great!

Although FWIW, you don't need certbot. Dehydrated does pretty much the same thing (i.e. they're both ACME clients which get certificates from Let's Encrypt).

Evan's picture

Evan - Wed, 2019/10/30 - 17:48
I don't see the CA or CA_Terms lines in this file. Should they be added? or do they need to be modified elsewhere?
Jeremy Davis's picture

Jeremy Davis - Wed, 2019/10/30 - 22:21

Yes add them in.

FWIW Dehydrated has built in defaults for a number of settings, so adding new values (such as CA and CA_TERMS) in the config file will override the builtin defaults.

Jeremy Davis's picture

Jeremy Davis - Tue, 2019/10/29 - 23:50

Ryan (via support) has just provided me with a (possible) short term workaround for the issue related to failures with multiple domains configured. I have posted it to the relevant GH issue, but will give a brief overview here.

Essentially he suggests doing the update (as noted above by Igor or on issue #1359). Once that has been configured, reduce the domains that you are requesting certs for to one and run the wrapper script. Then re-add the other domains and re-run the wrapper script.

I haven't tested it myself, but thought it worth sharing. If anyone else tests this out, please post back with feedback.

Unfortunately, I still haven't had a chance to complete the work I've started to provide a "proper" fix for #1360, but fingers crossed, I'll have the final piece for that soon. If anyone is willing and able to help out with that, please let me know.

Max Stummer's picture

Max Stummer - Sat, 2019/11/09 - 01:18
Hey guys! I had the same issues as described above. I followed Igor's instructions to update dehydrated, but am now getting a new error: 
>> /usr/bin/dehydrated --register --accept-terms

+ Account already registered!

>> /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper

[2019-11-08 22:57:24] dehydrated-wrapper: INFO: started
[2019-11-08 22:57:25] dehydrated-wrapper: INFO: found apache2 listening on port 80
[2019-11-08 22:57:25] dehydrated-wrapper: INFO: stopping apache2
[2019-11-08 22:57:25] dehydrated-wrapper: INFO: running dehydrated
  + ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/acme/new-acct (Status 400)

Details:
HTTP/2 400
server: nginx
date: Fri, 08 Nov 2019 22:57:29 GMT
content-type: application/problem+json
content-length: 134
cache-control: public, max-age=0, no-cache
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
replay-nonce: 010174sP_CDDsphjq--BZ_XvnkZ5xQvQjwR_b2zJ5xmbhVI

{
  "type": "urn:ietf:params:acme:error:accountDoesNotExist",
  "detail": "No account exists with the provided key",
  "status": 400
}

/etc/dehydrated/confconsole.hook.sh: line 54: request_failure: command not found
[2019-11-08 22:57:29] dehydrated-wrapper: FATAL: dehydrated exited with a non-zero exit code.
[2019-11-08 22:57:29] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert & key.
[2019-11-08 22:57:29] dehydrated-wrapper: INFO: starting apache2
[2019-11-08 22:57:29] dehydrated-wrapper: INFO: starting stunnel4
[2019-11-08 22:57:29] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.
Note the 'account already registered' vs 'no account exists' errors. Any help much appreciated! Cheers, Max
Jeremy Davis's picture

Jeremy Davis - Sat, 2019/11/09 - 02:55

TBH, I'm not 100% sure, but I think that perhaps it's the switch from the v1 API to the v2 API. I think it's worth trying to clear Dehydrated's data. I.e. try this:

mv /var/lib/dehydrated /var/lib/dehydrated.bak

And then retry:

/usr/bin/dehydrated --register --accept-terms
/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper

If that still doesn't work, could you please show me the contents of your config file:

cat /etc/dehydrated/confconsole.config
Max Stummer's picture

Max Stummer - Sat, 2019/11/09 - 23:52
Hey Jermey! Thanks for your reply, it got me one step further: just renaming/removing that directory was breaking other stuff (different 'folder does not exist' errors later). what got me one step further was reinstalling dehydrated and removing that folder: 
apt-get remove dehydrated
mv /var/lib/dehydrated /var/lib/dehydrated.bak
apt-get -t stretch-backports install dehydrated
/usr/bin/dehydrated --register --accept-terms
That worked ok to create the account. Unfortunately, now running dehydrated wrapper gives me the following error: 
/usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper
[2019-11-09 21:41:27] dehydrated-wrapper: INFO: started
[2019-11-09 21:41:28] dehydrated-wrapper: INFO: found apache2 listening on port 80
[2019-11-09 21:41:28] dehydrated-wrapper: INFO: stopping apache2
[2019-11-09 21:41:29] dehydrated-wrapper: INFO: running dehydrated
/etc/dehydrated/confconsole.hook.sh: line 54: this_hookscript_is_broken__dehydrated_is_working_fine__please_ignore_unknown_hooks_in_your_script: command not found
[2019-11-09 21:41:30] dehydrated-wrapper: FATAL: dehydrated exited with a non-zero exit code.
[2019-11-09 21:41:30] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert & key.
[2019-11-09 21:41:30] dehydrated-wrapper: INFO: starting apache2
[2019-11-09 21:41:30] dehydrated-wrapper: INFO: starting stunnel4
[2019-11-09 21:41:30] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.
Any ideas on your end? Thanks for your help, Max
Jeremy Davis's picture

Jeremy Davis - Sun, 2019/11/10 - 22:05

Judging from the error message "this_hookscript_is_broken__dehydrated_is_working_fine__please_ignore_unknown_hooks_in_your_script", I'm guessing that you haven't updated the hook script. To do that (as per step 2 of the workaround noted on the issue):

GH_URL=https://raw.githubusercontent.com/turnkeylinux/confconsole/master
GH_HOOK=share/letsencrypt/dehydrated-confconsole.hook.sh
CC_HOOK="$DEHYD_ETC/confconsole.hook.sh"
SH_HOOK=$SHARE/dehydrated-confconsole.hook.sh

wget $GH_URL/$GH_HOOK -O $SH_HOOK
cp $SH_HOOK $CC_HOOK

Hopefully that should get you up and running...

Jeremy Davis's picture

Jeremy Davis - Tue, 2019/11/12 - 08:08

Please note that I've just published Confconsole v1.1.1. It's not yet available from our repos (although will be soon) and still requires some specific steps to install and set up on a v15.x server (although better than instructions published previously).

Please note that users who have already updated via various other means are still recommended to install this update as it includes reliability fixes for add-water; our custom challenge mini-server. Please see the release notes for full step by step setup and further info - instructions cover both new and existing users.

Any issues, please ask. Any feedback (e.g. anything that isn't clear, etc). Please ask.

Jeremy Davis's picture

Jeremy Davis - Wed, 2019/12/04 - 05:50

It's been bought to my attention that after installing the v1.1.1 Confconsole update, the add-water service is being inadvertently enabled. That means that on reboot, it will start up and will likely block Apache (or other webserver) from starting!

The fix is easy:

systemctl disable add-water

Please note that on any server where you have already run the v1.1.1 update, you need to apply the above line and there is no value in updating to the newer package. For any new servers you launch though (or for anyone else who hasn't applied any fixes and stumbles across this thread) the newer v1.1.2 release resolves this issue (it's exactly that same as v1.1.1, but doesn't auto enable the add-water service on install).

Add new comment