Andi Northrop's picture

Hi there, I posted this over in the confconsole docs as a comment, but I guess the forums are better monitored? Anywho, to recap:

OK, I may be missing something obvious but I can't see what I should be doing after the first bullet point to make dehydrated pick up the additional domains and run the wrapper to get certificates for those new domains?

Will it just pick up the changes when the daily cron runs or do I need to invoke it myself somehow?

Cheers for any help!

Since then I've figured that I can call dehydrated from the terminal (duh) but I can't get it working.

To start with dehydrated by itself looks for domains.txt (instead of confconsole.domains.txt), a tweak to dehydrated's config file sorts that but now I'm getting a challenge error of 403:

Processing resources.citygatechurch.net
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for resources.citygatechurch.net...
 + Responding to challenge for resources.citygatechurch.net...
ERROR: Challenge is invalid! (returned: invalid) (result: {
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:acme:error:unauthorized",
    "detail": "Invalid response from http://resources.citygatechurch.net/.well-known/acme-challenge/Th_2SY-ekF4kPgs1Ea-UHpttUjM_80bifGs_bLcQw-0: \"\u003c!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\"\u003e\n\u003chtml\u003e\u003chead\u003e\n\u003ctitle\u003e404 Not Found\u003c/title\u003e\n\u003c/head\u003e\u003cbody\u003e\n\u003ch1\u003eNot Found\u003c/h1\u003e\n\u003cp\"",
    "status": 403
  },

Any ideas? I'm wary of trying too many times without advice lest I be blocked by letsencrypt.

This all runs fine from Confconsole but I've got new domains I need to add in so I need to get it sorted this way (I'm running 14.2 LAMP).

Cheers!

Forum: 
Andi Northrop's picture

Having rummaged around in the dehydrated docs, my best guess is that I need to add an alias to Apache in the alias.conf mod to the tune of:

Alias /.well-known/acme-challenge /var/www/dehydrated

Am I on the right lines?

Also, if I change dehydrated settings will that break the Confconsole method (not a major issue now but I may wish to use it in the future)?

Andi Northrop's picture

That Alias should probably read:

Alias /.well-known/acme-challenge /var/lib/dehydrated/acme-challenges

 

Jeremy Davis's picture

It's been a little while since I worked on it, but IIRC you should just need to add your additional domains to /etc/dehydrated/confconsole.domains.txt. Each new domain goes on a separate line, with up to 4 sub-domains. E.g.:
example.com www.example.com blog.example.com
example.net www.example.net forums.example.net
domain.com www.domain.com
The last in the list should write to /etc/ssl/private/cert.pem so will also be the cert that will work for Webmin and Webshell.

You can then run it manually (to check it works) by directly invoking the TurnKey dehydrated-wrapper:

/usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper

If you want to do some testing without risking getting blocked by Let's Encrypt, please add the staging server to your conf file (/etc/dehydrated/confconsole.config). Make sure you remove it when you're done testing though.

Want some info about it? Use the --help switch. Want more info about what it's doing? Try the --log-info switch or if you want really verbose output, set DEBUG. I.e.

export DEBUG=y
/usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper
If you use the Confconsole dehydrated-wrapper to launch dehydrated, you won't need to worry about setting aliases etc as it uses it's own mini-webserver to serve the challenges.

If you don't like that long path and want to make dehydrated-wrapper a "command"; then add a symlink to /usr/local/bin:

ln -s /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper /usr/local/bin/dehydrated-wrapper
Then you should be able to launch it simply by typing "dehydrated-wrapper".

Once it's done it's thing, then you should find your certificates in /etc/ssl/private/DOMAIN correction: /var/lib/dehydrated/certs/DOMAIN, e.g. using my domain list above, I should have:

/etc/ssl/private/example.com/
/etc/ssl/private/example.net/
/etc/ssl/private/domain.com/

/var/lib/dehydrated/certs/example.com/
/var/lib/dehydrated/certs/example.net/
/var/lib/dehydrated/certs/domain.com/

TBH, I didn't exhaustively test with multiple domains, so there is a chance that I missed something.

If you want to keep going the way you are, then it's possible, but you'll need to write your own hook script to write out the certificates that it gets. The hook script that ships with confconsole is designed to integrate with the bundled mini-server.

Regardless, it seems clear that we either need to improve the docs a bit (giving a step-by-step walkthrough) and/or make confconsole support multiple domains...

Andi Northrop's picture

Thanks Jeremy, that makes more sense.

It runs now but it's only writing to /etc/ssl/private/cert.pem meaning that only the domain (and subs) on the last line of confconsole.domains.txt is getting a certificate.

It's not writing any of the extra domains to directories e.g. /etc/ssl/private/example.com/

/var/lib/dehydrated/certs/ contains directories for each domain but nothing in /etc/ssl/private

Andi Northrop's picture

Hi Jeremy, just wondering if there's been any progress on this?

Cheers, Andi

Jeremy Davis's picture

Sorry for the radio silence. I've been tied up trying to finalise v14.2 and get the basics in place for v15.0. I'm pretty close, but still some minor issues.

I've fairly sure that this should be a pretty easy fix. I just need to put an hour or 2 aside and sit down with it.

I can't promise, but I'll try to get you something to test this week.

Please do not hesitate to bump this if your curious how I'm going.

Jeremy Davis's picture

Deep apologies on the delay on this.

But you know what?! It works as expected for me!?

Reading back through your posts, the issue is the path that you're looking for the certificates in! I'm really sorry that I didn't spot that sooner. As per the docs, please check /var/lib/dehydrated/certs/ for the specific per domain certificates. I'll replay what I did to confirm it's working as expected:

Double check DNS is configured right (from my desktop):

user@ninjux ~$ dig jeremydavis.org lamp.jeremydavis.org lamp-tester.tklapp.com +short
52.65.87.99
52.65.87.99
52.65.87.99

That looks good! So ready to run the dehydrated-wrapper on my lamp server. First, my confconsole.domains.txt:

root@lamp ~# cat /etc/dehydrated/confconsole.domains.txt

# please use this file with confconsole or
# alternatively use dehydrated with it's appropriate
# configuration directly
jeremydavis.org lamp.jeremydavis.org
lamp-tester.tklapp.com

Now to run dehydrated. You shouldn't need the --force switch, but I do as I've already had this working and want to demonstrate the full process:

root@lamp ~# dehydrated-wrapper --force
[2017-11-28 23:22:00] dehydrated-wrapper: INFO: started
[2017-11-28 23:22:00] dehydrated-wrapper: INFO: found apache2 listening on port 80
[2017-11-28 23:22:00] dehydrated-wrapper: INFO: stopping apache2
[2017-11-28 23:22:01] dehydrated-wrapper: INFO: running dehydrated
[2017-11-28 23:22:13] confconsole.hook.sh: SUCCESS: Cert request successful. Writing cert.pem & cert.key for jeremydavis.org to /etc/ssl/private
[2017-11-28 23:22:19] confconsole.hook.sh: SUCCESS: Cert request successful. Writing cert.pem & cert.key for lamp-tester.tklapp.com to /etc/ssl/private
[2017-11-28 23:22:19] dehydrated-wrapper: INFO: dehydrated complete
[2017-11-28 23:22:19] dehydrated-wrapper: INFO: Cleaning backup cert & key
[2017-11-28 23:22:19] dehydrated-wrapper: INFO: starting apache2
[2017-11-28 23:22:20] dehydrated-wrapper: INFO: starting stunnel4
[2017-11-28 23:22:20] dehydrated-wrapper: INFO: dehydrated-wrapper completed successfully.
Now check in /var/lib/dehydrated/certs for the certificates:
root@lamp ~# tree /var/lib/dehydrated/certs
/var/lib/dehydrated/certs
|-- jeremydavis.org
|   |-- cert-1511911322.csr
|   |-- cert-1511911322.pem
|   |-- cert.csr -> cert-1511911322.csr
|   |-- cert.pem -> cert-1511911322.pem
|   |-- chain-1511911322.pem
|   |-- chain.pem -> chain-1511911322.pem
|   |-- fullchain-1511911322.pem
|   |-- fullchain.pem -> fullchain-1511911322.pem
|   |-- privkey-1511911322.pem
|   `-- privkey.pem -> privkey-1511911322.pem
`-- lamp-tester.tklapp.com
    |-- cert-1511911333.csr
    |-- cert-1511911333.pem
    |-- cert.csr -> cert-1511911333.csr
    |-- cert.pem -> cert-1511911333.pem
    |-- chain-1511911333.pem
    |-- chain.pem -> chain-1511911333.pem
    |-- fullchain-1511911333.pem
    |-- fullchain.pem -> fullchain-1511911333.pem
    |-- privkey-1511911333.pem
    `-- privkey.pem -> privkey-1511911333.pem

2 directories, 20 files
Looks like everything is in there. So all should be well. Although you'll still need to configure the individual webserver sites to use the relevant certificates. We haven't automated that because it will depend on the webserver being used and how you are configuring the sites. But it should be relatively straight forward if you understand how that works (it's is after all "advanced" functionality of confconsole).

Rereading the previous conversation again, I think what introduced your confusion is the log message "Writing cert.pem & cert.key for xyz.com to /etc/ssl/private".

FWIW, the docs do mention that the last cert to be written will be written there (and it is). Whats actually happening is that each cert is being written there, so each one overwrites the previous. In retrospect, I perhaps should have considered how I might make the log messages a bit clearer when multiple domains are used.

TBH, it was a last minute addition to support more complex use cases, and it wasn't implemented as well as it could have been. Also as I've noted, the docs aren't as clear as they should be. I'll spend a little more time on this and try to tidy it up a bit.

If you have any specific feedback on might be best, please let me know.

Jeremy Davis's picture

Let's Encrypt have updated their user agreement, and for it to work, it needs to be pointing to the latest version of that. The fix is easy, have a look here.

I'll have a look at improving the log messages and we'll hopefully be updating confconsole soon.

Jeremy Davis's picture

If you don't want to have them set as separate webserver sites (i.e. virtual hosts) then you can actually create a single certificate for all domains and subdomains! All you need to do is put them all on the one line of confconsole.domains.txt. E.g.:
root@lamp ~# cat /etc/dehydrated/confconsole.domains.txt

# please use this file with confconsole or
# alternatively use dehydrated with it's appropriate
# configuration directly
jeremydavis.org lamp.jeremydavis.org lamp-tester.tklapp.com
root@lamp ~# dehydrated-wrapper --force
[2017-11-28 23:50:23] dehydrated-wrapper: INFO: started
[2017-11-28 23:50:23] dehydrated-wrapper: INFO: found apache2 listening on port 80
[2017-11-28 23:50:23] dehydrated-wrapper: INFO: stopping apache2
[2017-11-28 23:50:25] dehydrated-wrapper: INFO: running dehydrated
[2017-11-28 23:50:41] confconsole.hook.sh: SUCCESS: Cert request successful. Writing cert.pem & cert.key for jeremydavis.org to /etc/ssl/private
[2017-11-28 23:50:41] dehydrated-wrapper: INFO: dehydrated complete
[2017-11-28 23:50:41] dehydrated-wrapper: INFO: Cleaning backup cert & key
[2017-11-28 23:50:41] dehydrated-wrapper: INFO: starting apache2
[2017-11-28 23:50:42] dehydrated-wrapper: INFO: starting stunnel4
[2017-11-28 23:50:42] dehydrated-wrapper: INFO: dehydrated-wrapper completed successfully.
And here is the certs directory (but this time they're all written to /etc/ssl/private). Note that I deleted all the directories that were there initially so this is new for this run of dehyrdated-wrapper.
root@lamp ~# tree /var/lib/dehydrated/certs #
/var/lib/dehydrated/certs
`-- jeremydavis.org
    |-- cert-1511913025.csr
    |-- cert-1511913025.pem
    |-- cert.csr -> cert-1511913025.csr
    |-- cert.pem -> cert-1511913025.pem
    |-- chain-1511913025.pem
    |-- chain.pem -> chain-1511913025.pem
    |-- fullchain-1511913025.pem
    |-- fullchain.pem -> fullchain-1511913025.pem
    |-- privkey-1511913025.pem
    `-- privkey.pem -> privkey-1511913025.pem

1 directory, 10 files
Andi Northrop's picture

Thanks Jeremy, I haven't had time to look at this today but thanks for digging into it. Hopefully this sorts us out!

Jeremy Davis's picture

You're most welcome.

FWIW I'll be looking to refactor the confconsole code to allow users to create a single certificate with up to 5 separate domains (be they sub domains, or completely separate domains). I'm not sure how long it will take, but I plan to work on that over the coming week or so. Unfortunately you may need top make do with what we have for now, but hopefully we'll have something better soon!

You will still need to manually adjust the confconsole.domains.txt if you want multiple certificates. But hopefully that will make it more useful. There's generally no need to create separate certificates, even if you are serving the sites completely separately. I.e. by using named based virtual hosts.

I have the certs generated but not sure which of the file si ma supposed to point to ?

 

ssl_certificate /var/lib/dehydrated/certs/ydexchange.youdopet.com/cert.pem;
        ssl_certificate_key /var/lib/dehydrated/certs/ydexchange.youdopet.com/privkey.pem;

Is this correct

Thanks


Jeremy Davis's picture

Although TBH, I'm not sure OTTOMH. The first test is if it works! :)

Although actually, I just did a quick google and apparently it should be like this:

ssl_certificate /usr/local/etc/dehydrated/certs/example.com/fullchain.pem;

ssl_certificate_key /usr/local/etc/dehydrated/certs/example.com/privkey.pem;
Hope that helps.

I am trying to run to renew the certs in etc/dehydrated/confconsole.domains.txt

root@nginx-php-fastcgi ~# /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper
[2018-10-08 16:06:13] dehydrated-wrapper: INFO: started
[2018-10-08 16:06:13] dehydrated-wrapper: INFO: found nginx: listening on port 80
[2018-10-08 16:06:13] dehydrated-wrapper: FATAL: An unexpected service is listening on port 80: nginx:
[2018-10-08 16:06:13] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert & key.
[2018-10-08 16:06:13] dehydrated-wrapper: INFO: starting stunnel4

Any idea how to resolve ?


Jeremy Davis's picture

It was a bug in confconsole. Updating to the latest package (1.1.0+2+g6c2aad9) should fix it.

I ran 
apt-get update
apt-get dist-upgrade

and also

apt-get update
apt-get install confconsole python-bottle authbind
apt-get install -t jessie-backports dehydrated

get

dehydrated is already the newest version.

But still get same error when running /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper

Thanks

Paul

 


Jeremy Davis's picture

I'm guessing from the commands that you've used it's a v14.x appliance?! Otherwise you'd get an error when trying to install from jessie-backports (v15.x is based on Stretch; v14.x was based on Jessie).

You can check what TurnKey version like this:

turnkey-version

Assuming you are running v14.x, then your issue is different to this one. The v14.2 issue is documented here, you'll also find the workaround documented there in that thread too. You'll need to manually apply it.

I hope that helps.

Yes running 14.2 however I already applied that fix a while ago.

confconsole.config is as follows

BASEDIR=/var/lib/dehydrated
WELLKNOWN="${BASEDIR}/acme-challenges"
DOMAINS_TXT="/etc/dehydrated/confconsole.domains.txt"
HOOK="/etc/dehydrated/confconsole.hook.sh"

LICENSE="https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf"

Whats best option update turnkey ?


Jeremy Davis's picture

Can you please post the exact error message you're getting?

Re updating TurnKey, now matter waht, make sure you have a tested backup before you start. Then you essentially have 2 options:

  • an "in place" upgrade like you would on any other Debian install (TurnKey is Debian under the hood); or
  • use TKLBAM to migrate your data to a new v15.0 appliance

Either way it may require some manual config adjustment to get everything working nicely.

Personally I prefer the latter as then you can leave your original server untouched and do a data restore to a new v15.0 server. I'd recommend doing that in a local VM. Worst case scenario, you can trash it and start again. There isn't yet a v15.0 doc page, but much of the v14.x page still applies (although obviously not the specifics). I would anticipate the migration to v15.0 being more straight forward than v13.x (or earlier) was to v14.x.

If you'd like to share a bit more about your specific server, I'm happy to provide some opinion. And/or if you hit any bumps, please ask.

I get this error when I run

/usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper

root@nginx-php-fastcgi ~# /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper
[2018-10-17 14:36:16] dehydrated-wrapper: INFO: started                                                                          
[2018-10-17 14:36:16] dehydrated-wrapper: INFO: found nginx: listening on port 80                                                
[2018-10-17 14:36:16] dehydrated-wrapper: FATAL: An unexpected service is listening on port 80: nginx:                           
[2018-10-17 14:36:16] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert & key.                          
[2018-10-17 14:36:16] dehydrated-wrapper: INFO: starting stunnel4 

and I have already applied the fix to domains.txt  per version 14.2

 

 

Ran

apt-get install confconsole

and I get newest version message


root@nginx-php-fastcgi ~# apt-cache policy confconsole
confconsole:                                                                                                                     
  Installed: 1.0.1+4+g7e2bdbe                                                                                                    
  Candidate: 1.0.1+4+g7e2bdbe                                                                                                    
  Version table:                                                                                                                 
 *** 1.0.1+4+g7e2bdbe 0                                                                                                          
        999 http://archive.turnkeylinux.org/debian/ jessie/main amd64 Packages                                                   
        100 /var/lib/dpkg/status                                                                                                 
root@nginx-php-fastcgi ~#


So it looks like I have wrong package. How do I update ?

Thanks


Jeremy Davis's picture

Ah that sucks! It looks like you are running a v14.x server. The newer version of Confconsole has only been built for v15 as I didn't realise that this bug also affects the v14.x build.

I assumed that it was a gap in my testing that caused the issue, but your experience suggests that it may have actually been a Debian update somewhere along the line that broke it (that was also applied to Debian 8/Jessie).

TBH, I'm not sure if/when we'll be able to release an update to the Confconsole version in v14.x. There are non-backwards compatible changes made to the package for the v15.x release, so we'd need to branch the code to remove those breaking changes. Also Alon needs to build the packages to be able to publish them (he has the key to the apt repo) but I'm pretty sure that he no longer has a v14.x package build env anymore (hence why we haven't released a v14.x update for a while). Obviously if there were a serious security issue, we'd do what needed to be done to resolve it, but for a bug like this it's harder to justify pulling resources from v15.x ongoing development and maintenance to address it.

So that leaves you with a few options:

  1. Migrate your data to a new v15.x server using TKLBAM.
  2. Upgrade the base OS to Debian 9/Stretch - I.e. Debian "in-place" upgrade.
  3. Manually patch the Confconsole code.

3 would likely be the quickest and easiest. But depending on what you are running on your server and where, for the longer term 1 or 2 might be better options?

Regarding option 1, there isn't much documentation about migration from v14.x to v15.x (yet) but it should be generally along the lines of previous major version migrations. I.e. the general workflow of this doc should still apply (although not the specifics). My experience would suggest that there have been less breaking changes between Debian 8/Jessie (basis of v14.x) and Debian 9/Stretch (basis of v15.x) so (at least in theory) it should be less painful than migrating from v13.x to v14.x. YMMV though. If you take that path, I'd love to hear how it goes and would be more than happy to help with any issues you hit. We could make your experiences a basis for a v15.x migration doc page. If so, please start a new thread for that though.

Option 2 may be preferable in some scenarios, and for all intents and purposes, will give you something akin to v15.x (although not quite as not all our tweaks are packaged). Whilst that may be a workable option, personally, I'd still be inclined to use TKLBAM to migrate your existing data to a fresh (v14.x) VM for the purposes of testing the upgrade first. That way if something goes seriously pearshaped, your current server would remain unharmed.

As hinted above, option 3 is almost undoubtedly the quickest and easiest fix. IMO it's highly unlikely to cause any future issues, although it is a little hacky. Worst case, reinstalling the current version of Confconsole should bring you back to where you are. The change required is contained in this commit, although I made an additional minor adjustment here which is probably worth including.

So to apply that to your package, you just need to add this code to the existing sed statement (i.e. just before the closing single quote at the end of the existing line):

; s|[^a-zA-Z0-9]||g

So that the final line looks like this:

netstat -ltpn | grep ":80 " | head -1 | cut -d/ -f2 | sed -e 's|[[:space:]].*$||; s|[^a-zA-Z0-9]||g'

(I split it over 2 lines in the code to keep the line length within limits which makes it easier to read, but strictly speaking that's not required). You'll find the file that needs editing here: /usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper.

I hope that helps. If you have more questions, or have any issues, please post back.

FWIW, in the longer term, we plan to provide an additional TurnKey apt repo which will either have automatically built packages, and/or allow me direct push access to it. Whilst it will require users to manually enable/add an additional TurnKey repo, it would make the workflow for producing updates for things like this much easier (for both us and users such as yourself). We hope to have that in place for our v15. apt repository before we move to a Debian 10/Buster base. But whether that will be backported to v14.x/Debian 8/Jessie remains to be seen.

That has fixed it. FYI its line 113 I commente the origianl and pasted in

netstat -ltpn | grep ":80 " | head -1 | cut -d/ -f2 | sed -e 's|[[:space:]].*$||; s|[^a-zA-Z0-9]||g'

Thanks


Jeremy Davis's picture

Thanks for confirming the fix.

Add new comment