Jeff Wilson's picture
Hi, are there any docs that I can follow that will help me replace the reverse proxy nginx self signed cert in an app appliance like (the new) mattermost? I've looked and I can't find anything. Any helpful direction would be very appreciated! Thanks!
Forum: 
Jeremy Davis's picture

First up, it's probably being aware that TurnKey is based on Debian, so often instructions that apply to Debian should apply to TurnKey as well. However, in the case of SSL certs that's not quite the case as we do tweak things a bit there.

The default certificate that all SSL TurnKey services use (e.g. webservers, Webshell & Webmin) can be found at /etc/ssl/private/cert.pem (and the key at /etc/ssl/private/cert.key). They are put together is a very specific way so that they work with all the different services that may use the same certs.

If you wish to use free Let's Encrypt certificates, then Confconsole has a Let's Encrypt plugin that can do that for you. That will replace the default certs that I noted above and the same cert should work with all HTTPS endpoints on your server. Note that it will only work if your domain already resolves to your server IP and port 80 needs to be publicly accessible. It's perhaps also worth noting, that this should "just work" in v16.0+ appliances, but older v15.x appliances have a number of issues. There are fixes for the issues, but you are advised to use the v16.0 appliance if possible.

Alternatively, if you have a thrid party cert that you'd like to use, so long as they are the correct format and you put them together the same as the default ones, you can replace those and all the services should leverage the updated cert and SSL should "just work" without browser warnings.

However, if you have third party certs and are in a rush and just want to replace the certificate (and key) for Nginx then that's possible too. You can put your third party certificates wherever you want (although /etc/ssl/private is a good place IMO) and then update the paths in /etc/nginx/include/ssl by default they are:

ssl_certificate      /etc/ssl/private/cert.pem;
ssl_certificate_key  /etc/ssl/private/cert.key;

I hope that helps...

Jeff Wilson's picture

So I have replaced the .pem and .key files with what my internal company cert process gives me. Mattermost is no longer accessible and shows it can't be reached. Are there any details about what the third party cert format should be? Mine is:
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-,F59315
-----END RSA PRIVATE KEY-----
  I'm able to back out of the cert changes and mattermost is then accessible after a reboot.  Any tips on where I can look?  I see from nginx's logs that it doesn't start:  
Jul 08 19:24:35 mattermost nginx[940]: Enter PEM pass phrase:
Jul 08 19:24:35 mattermost nginx[940]: nginx: [emerg] SSL_CTX_use_PrivateKey_file("/etc/ssl/private/cert.key") failed (SSL: error:2807106B:UI routines:UI_process:processing error:while reading strings error:0906406D:PEM routines:PEM_def_ca
Jul 08 19:24:35 mattermost nginx[940]: nginx: configuration file /etc/nginx/nginx.conf test failed
Jul 08 19:24:35 mattermost systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
  nginx.conf looks pretty geneneric and I can't see any issues there (and I haven't changed anything too).  any help you can provide woudl be fantastic.  I've read through the 2018 post here as well:   https://www.turnkeylinux.org/blog/use-cer-or-p7b-certs-with-turnkey
Jeremy Davis's picture

TBH, it's not something that I'm super familiar with. Without having a real key to play with, I'm really just relying on my limited understanding, combined with the online resources that I've checked up on. So it's pretty much the blind leading the blind here, but let's see how we go...

First up, it appears that the private key part appears to be encrypted. And according to the output you've posted, it also has a password. Note where the Nginx log says "Enter PEM pass phrase:".

Also, I'm not 100% sure, but a bit of reading that I've done suggests that your key might be in a legacy format that may or may not work with Nginx and/or Stunnel (Webmin/Webshell are behind Stunnel). I'm judging that by the "BEGIN RSA PRIVATE KEY/END RSA PRIVATE KEY" text. It may be fine, but if not, you may need to convert it to a "pkcs#8" format key (which will have have the text "BEGIN PRIVATE KEY/END PRIVATE KEY" text instead).

It's also worth noting that Nginx requires the key section to be in a separate file. If you want to use the same cert/key with Stunnel (Webmin/Webshell) too, then IIRC you'll want them altogether in the cert file, plus just the key section in the key file. That way Nginx will ignore the key section in the cert file and read it from the key file and Stunnel will read both the key (and the cert) from the cert file. I.e. have your files like this:

/etc/ssl/private/
|-- cert.key    # Just the "PRIVATE KEY" section
|-- cert.pem    # the whole thing (inc the "PRIVATE KEY" section)
|-- dhparams.pem
`-- ssl-cert-snakeoil.key

So just considering Nginx and the password protected key, you essentially have 3 options, either way you'll need the password:

  1. Don't use the Nginx service and instead manually launch Nginx and manually give the password when requested. The most secure but least practical option IMO.
  2. Use the Nginx "ssl_password_file" directive - note the path to a text file that includes the password. Less secure, although if you set ownership to root and permissions to be 600 (owner read/write; nothing for group and everyone) then it's only readable by the root user (untested, but Nginx should start as root; allowing it to read the file, then fork as www-data user so no further actions should have access).
  3. Remove the password altogether. Least secure but most common configuration. So long as the key & cert are owned by root and no access to other users (as per the password file in option 2), then it should be "secure enough" and more-or-less as secure as option 2. This is the option I'd choose personally (and IMO essentially the "industry standard") - IMO if someone has access to your server all bets are off regardless...
  4. If you choose option 1 or 2 above; there is a chance that you may not need to do anything more (than the appropriate action required by each action). If you want to follow option 3, then OpenSSL should be able to do what is required. You may be able to pass the file that you have into OpenSSL, although I suspect that you will need to copy out the "RSA PRIVATE KEY" section, i.e. these lines:

    -----BEGIN RSA PRIVATE KEY-----
    Proc-Type: 4,ENCRYPTED
    DEK-Info: DES-EDE3-,F59315
    -----END RSA PRIVATE KEY-----
    

    (And all the lines of random characters that form the key between these)

    From what you've posted, assuming that the filename is cert.pem and the file(s) are in /path/to/where/your/keys/are; this should do the trick to separate the bulk of the file from the key:

    cd /path/to/where/your/keys/are
    sed -n '/-----BEGIN RSA PRIVATE KEY-----/,$p' cert.pem > enc.key
    sed '/-----BEGIN RSA PRIVATE KEY-----/Q' cert.pem > new.cert
    

    That will leave you with the original file(s), plus two new files; enc.key & new.cert. To decrypt the key file, removing the password and converting to pkcs8 format (just in case), this should do the trick:

    openssl pkcs8 -topk8 -nocrypt -in enc.key -out dec.key

    You will be asked for the password interactively, so you'll need to enter it when asked. Alternatively, you could include it in the command, via the "-passin" switch, like this (assuming that your password is MY_PASS).

    openssl pkcs8 -topk8 -nocrypt -in enc.key -passin pass:MY_PASS -out dec.key

    Double check that the new key file (dec.key) no longer includes the note about encryption with cat. I.e.:

    cat dec.key

    Then assuming that you want them bundled back together and rename them to the default names; do this:

    mv dec.key /etc/ssl/private/cert.key
    mv new.cert /etc/ssl/private/cert.pem
    cat /etc/ssl/private/cert.key >> /etc/ssl/private/cert.pem
    cat /etc/ssl/private/dhparams.pem >> /etc/ssl/private/cert.pem
    chown root:root /etc/ssl/private/cert.pem /etc/ssl/private/cert.key
    chmod 400 /etc/ssl/private/cert.pem /etc/ssl/private/cert.key
    

    And if I've read everything correctly (fingers crossed), you should now be all good now... :)

    If not, please let me know which step causes issues and the exact error message/messages you get.

Add new comment